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ABSTRACT: 

The present invention provides a method for editing digital audio 
information, such as musical material. Original musical parameters 
(302) are extracted and/or inputted from recorded original digital 
audio material (300) . The original musical parameters (302) are then 
edited. The resulting edited musical parameters (304) are compared to 
the original musical parameters (302) to provide time varying control 
functions (308, 310, 312) . The original digital audio material (300) 
is then processed with signal processing algorithms (314, 316, 318) 
which are controlled by the time varying control functions (308, 310, 
312) . This processing changes the original digital audio material 
(300) into new digital audio material (320) having musical 
characteristics which correspond to the edited musical parameters 
(304) . 

35 Claims, 24 Drawing figures 
Exemplary Claim Number: 1 
Number of Drawing Sheets: 16 
BRIEF SUMMARY: 

(1) CROSS REFERENCE TO RELATED APPLICATION 

(2) This application is a continuation-in-part of copending U.S. 
patent application Ser. No. 60/004,649 filed as a provisional 
application Sep. 29, 1995 in the names of Dan Timis and David Gerard 
Willenbrink under the title SYSTEM FOR EDITING DIGITAL AUDIO MATERIAL 
WITH MUSICAL PARAMETERS. This application claims priority from the 
prior provisional application. 

(3) COPYRIGHT NOTICE 

(4) A portion of the disclosure of this patent dociament contains 
material which is subject to copyright protection. The copyright 
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owner has no objection to the photographic reproduction by anyone of 
the patent document or the patent disclosure in exactly the form it 
appears in the Patent and Trademark Office patent file or records, 
but otherwise reserves all copyright rights whatsoever. 

(5) BACKGROUND OF THE INVENTION 

(5) The present invention is directed to a system for editing 
recorded and synthesized music, and more particularly to a computer 
program for transforming musically significant parameters of digital 
audio data. 

(7) Music is often recorded, produced and distributed as digital 
audio data. TVnalog audio signals from microphones, electric guitars, 
or other electronic instruments are converted into a series of 
digital samples that represent the instantaneous amplitude of the 
audio waveform. The digital signals are often immediately processed 
with digital reverberation, equalization and other transformations. 
Recorded samples can be stored on multi-track digital audio tape 
machines and computer mass storage systems. Separate digital files 
for vocals and instruments can be further processed and digitally 
mixed into a final master. The digital master can then be used to 
produce compact discs and other digital distribution media as well as 
analog distribution media such as audio cassettes. Compact discs 

(CDs) contain 16-bit samples that are sampled at the rate of 44,1 
kHz. 

^^^^^ ^ cTf Ilsyn tKe^i":2;e^Zs^ 

Sp^ecTalized personal computer software called s e quencer^^rio vT^nu sic 
to be composed in standard or special musical notation and played by 
sending sequences of signals to sound-producing equipment such as 
synthesizers. The Musical Instrument Digital Interface (MIDI) 
standard specifies the communications protocol that is used between 
devices that control performances (such as keyboards and sequencers), 
devices that produce sounds (such as synthesizers) , and devices that 
record and play back performances (such as digital audio tape 
recorders) . The product Vision, commercially available for several 
years from the present patent application's assignee Opcode Systems, 
Inc., is a popular sequencer program for personal computers. 

(9) Person^]^^computers are also used to provide CfXexibl^^^e^^ 
digl:Calz:auai5rma^e-r^^^ Multi-track recordings can be loaded onto the 
hard disk of a personal computer or directly recorded to the hard 
disk. Special software allows the playback of previously recorded 
tracks while additional tracks are recorded in a process known as 
overdubbing. Individual tracksjorj bhe fj ^shed material can be 
processed with specxa^ZEf^ecti^^^^e^ Cut, copy and 

paste operations can be used to produce a composite performance from 
a series of partial recordings with very high accuracy. Repetitive 



127 0 0 1 :4343 C: APPS ea cac e ea 200307241 15016541 m e 



C ie%20Ne 6 7 24 03 



DOCUMENT-DENTIFIER: US 5792971 A 



Page 4 of 47 



patterns such as drum rhythms can be automatically repeated, 

(10) Note sequences for synthesizers are often represented on the 
computer's monitor in piano roll format, where the X-axis represents 
time and the Y-axis represents the pitch of a note. (See FIG. 4b.) 
The length of a note in piano roll format represents its duration. 
Conventional music notation can also be used to represent sequences, 

(See FIG. 4a.) List windows represent musical parameters in numerical 
form allowing for very detailed editing, (See FIG, 4c,) 

(11) Digital editing programs usually represent sounds as 
waveforms, showing the instantaneous amplitude of the signal on a Y- 
axis with time on the X-axis. (See FIG, 5a.) This form of 
representation often shows the shapes of phrases and sometimes of 
individual notes especially with percussive sounds. However, editing 
on a note by note basis is not always easy, particularly for vocals 
and wind instruments, 

(12) The early versions of the product Studio Vision Pro (Version 
1,4 and Version 2.0 (both introduced before October 1994), also 
available from Opcode Systems, Inc., were the first commercial 
products to combine a MIDI sequencer with an editor for digital audio 
recordings. In this and competitive products, some tracks represent 
synthesized sounds (e.g., for drums and accompaniment), while other 
tracks on the screen show at the same time digital audio for other 
sounds in the same piece, such as vocals and solo instruments, 

(13) The professional recording and production of music is often a 
time-consuming and expensive process. The staff of a recording studio 
often records the same piece over and over until a perfect "take" is 
achieved. If there is one wrong note, an extraneous sound, a timing 
problem, a lack of synchronization with previously recorded tracks in 
a long song, or the like, it has often been necessary to abandon the 
entire take and start over. Dozens of "takes" are not uncommon, and a 
sizable well paid staff of musicians, recording engineers and 
producers is often involved. The significant capital equipment in a 
studio is also engaged during long recording sessions. The necessity 
for multiple "takes" accounts for a significant fraction of the costs 
of music recording. 

(14) Music production usually includes two distinct phases: 
recording and mixing. During recording, artists usually listen to 
previously recorded tracks played as performed and add additional 
"raw" tracks. The result is a multi-track recording. After the 
recording sessions are over and the artists have dispersed, a smaller 
staff often enhances the recorded material in a variety of ways and 
mixes the multiple tracks into two stereo tracks for duplication. 
Flaws in recorded material and new musical opportunities are often 
discovered during this production stage, when the original artists 
are usually no longer available. 
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(15) By contrast, sequencer music is stored in a form that is . 
easily editable and precisely reproducible. A variety of significant^ 
parameters are stored for every note. Live performances on keyboards 
and other MIDI controllers can be captured for latfir^p^Hi ting . Music 
can also be composed slowly, note by note and phrase by phrase and 
later edited to play at a designated tempo. Individual musical notes 
can be easily dragged on the screen to different pitches or durations 
and they can be assigned to different instruments. Defects in a take 
can be easily corrected and new musical ideas explored after the 
original recording. 

(16) The ease and flexibility of sequencers has had an enormous 
impact on the production of popular music, allowing individual 
composers and performers to produce complex and rich music working 
alone. Synthetic music, however, cannot always reproduce the nuances, 
complex timbres and ambiance of real instruments and cannot produce 
vocals at all. Accordingly, much music is still recorded in studios, 
with the producers selecting the best of many takes. 

(17) Musicians, recording engineers, and producers wish they could 
modify digital audio recordings with the same ease and flexibility 
only a MIDI sequencer can offer. The ability to rectify the pitch of 
only a few notes, to change durations, tempo, volume, and other 
significant musical parameters within a digital audio recording could 
significantly reduce the number of takes in a multi-track recording 
session. Many minor mistakes could be corrected and new musical ideas 
explored without requiring the presence of the original recording 
artists. Additionally, new musical effects previously impossible to 
produce could be available. 

(18) Changing the pitch of audio, modifying timing without altering 
the pitch, changing volume and filtering, are well understood digital 
signal processing techniques available in a few commercial programs. 
Two of the companies that have products offering time 
compression/expansion and pitch shifting are EMAGIC with the program 
"Logic Audio" and Steinberg with the application "Time Bandit." 
However, most often these features are applied globally on entire 
digital audio files or events. When it's desirable to change only a 
single note, finding that note and specifying all the parameters 
needed to be changed is a very tedious process. 

(19) Apart from allowing many musical parameters to be changed, 
sequencer programs offer musicians a familiar editing environment. 
Changing pitches and durations in a traditional notation 
representation, in a piano roll, or in a list window is a friendly 
and very intuitive process. By contrast, editing a waveform 
representation of digital audio is a rather difficult task. Apart 
from a few cases of percussive sounds, selecting a note from a stream 
of samples is a long and difficult process that involves a lot of 
trial-and-error iterations. 
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(20) Allowing digital audio to be represented in a form that is 
more familiar to the musician is a first step toward the goal of 
allowing digital recordings to be modified with the same ease as 
MIDI. A few commercially available computer programs or hardware 
devices (e.g.. Pitch to MIDI converters) offer the possibility to 
turn audio into MIDI information including note numbers, note on/off, 
volume, and pitch bend information. Among software products for 
personal computers, the program "Logic Audio" from EMAGIC offers 
"Audio to Score, " a feature that turns digital audio into musical 
notes. Another computer product that converts sound into MIDI is 
"Autoscore" from Wildcat Canyon Software. 

(21) Once digital audio information is represented in a musically 
significant manner, edits can be made to this representation in the 
familiar environments of traditional notation, piano roll, list 
window, or others. These changes can be automatically translated into 
parameters for Digital Signal Processing functions. By contrast, 
entering these parameters directly is a very tedious and non- 
intuitive process. These {SajrameteFis can then be used to control 
digital signal processing (-DS-P^) functions that will modify the 
recorded digital audio information resulting in new material that 
combines sonic qualities of the original audio with musically 
significant changes made through the yri-pi^ representation. Thus, the 
desiderata of edi^ixTg^dl^^gl-tSl— a^i^^^ 

a'ch-ie veti . 

(22) SUMMARY OF THE INVENTION 



(23) The present invention provides a method for editing digital 
audio information with music-like characteristics based on comparison 
of a first set of control codes associated with the source program 
and a second set of control codes preselected to represent a desired 
editorial change. The present i*iwent±rai^ for t^^a^ferm±ng»> 
musically significant paxam^1?Erszzo:f==^ information. Thus, 
generalized musical notation represented by digital information is 
used to edit the musical characteristics of the source audio program 
information to produce an edited audio program. 

(24) In accordance with the invention, original musical parameters 
are input or extracted from recorded original digital audio 
information. In one embodiment, the original musical parameters are 
edited. In an alternative embodiment, additional musical parameters, 
such as codes representing additional voicing and instrumentation can 
be introduced. The resulting edited musical parameters are compared \ 
to the original musical parameters to provide time varying control \ 
functions. The original digital audio information is then processed 1 
with digital signal processing (DSP) algorithms, which are controlleci 
by the time varying control functions. This processing changes the 1 
original digital audio information into new digital audio information 
having musical characteristics that correspond to tii^^ed^-te'cilln^Js^^^ 



127 0 0 1 :4343 C: APPS ea cac e ea 200307241 15016541 m e 



C ie%20Ne <5 7 24 03 



DOCUMENT-DENTIFffiR: US 5792971 A 



Page 7 of 47 



parameters. 

(25) The subject matter of this invention diclosure and the parent 
provisional patent application was first embodied in Studio Vision 
Pro Version 3.0 which was first commercially introduced in October 
1995 by Opcode Systems, Inc. The present disclosure merely restates 
the subject matter of the parent provisional application. 

(26) The invention will be better understood upon reference to the 
following detailed description in conjunction with the accompanying 
drawings . 

DRAWING DESCRIPTION: 

BRIEF DESCRIPTION OF THE DRAWINGS 

FIG. 1 illustrates an example of a computer system used to execute 
the software of the present invention. 

FIG. 2 shows a system block diagram of computer system 1 used to 
execute the software of the present invention. 

FIG. 3 shows a pictorial block diagram for an alternative real- 
time system used to implement the present invention. 

FIGS. 4a-4f illustrate various forms of musical representation as 
may be seen by a user in an interactive display or printout. 

FIGS. 5a-5c show various forms of graphical representation of 
digital audio material as may be seen in an interactive display or 
printout . 

FIG. 6 is a block diagram showing the fundamental process of the 
present invention . 

FIG. 7 is a. time domain graph showing an original digital audio 
waveform for a musical passage along with the waveform's 
corresponding representation in musical notation and piano roll 
format . 

FIG. 8 is a time domain graph showing an original digital audio 
waveform for a musical passage along with an edited representation in 
musicaX^^aotai J-on and piano roll format. ~ 



FIG. 9 is a time domain graph showing a time varying control 
function for time stretching along with representations in the 
original piano roll format and the edited piano roll format. 

FIG. 10 is a time domain graph showing a new digital audio 
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waveform for a musical passage along with the waveform's 
corresponding representation in musical notation and piano roll 
format based on an edited control function. 

FIG. 11 is a block diagram showing the process of extracting 
musical parameters from source digital audio information using DSP 
analysis functions guided by analysis control functions. 

FIG. 12 is a block diagram showing the process of editing control 
parameters for digital audio information. 

FIG. 13 is a block diagram illustrating the four phase process of 
one embodiment of the present invention; 

FIG. 14 is a block diagram showing the process of extracting 
musical parameters from an external model of digital audio 
information using DSP analysis functions. 

FIG. 15 is a block diagram showing the process of harmonizing 
musical parameters , 

FIG. 16 is a block diagram showing the process of modifying 
polyphonic source material. 

FIG. 17 is a general flow chart showing the process of the present 
invention . 

DETAILED DESCRIPTION: 

(1) DESCRIPTION OF SPECIFIC EMBODIMENTS 

(2) The present invention permits dl^rt^a^auxli^eo^bB^'^^ and 
enhanced with some of the flexibility already available in sequenced 
music. The method operates best on digital audio information having 
music-like characteristics such as pitch, timbre, cadence, time- 
dependent dynamics or like parameters that can be scored for 
subsequent reproduction. Voice and other audio signals often contain 
these music-like characteristics. In the context of music, the 
invention solves the long-standing problem of endless retakes by 
peooi^ting minor^fl aws in re cordin gs to be easily corrected, (jfan y— nevv^] 
/e^ear^v ^3uai"^ 

siu^Kou±2lE^e5^ . 

(3) Digital audio recordings can be represented in a form of 
notation that permits editing. The resulting edit changes are then 
applied to the audio recordings through digital signal processing 
techniques. A wide variety of parameters including pitch, timing, 
duration, loudness and timbre thus can be changed. The resulting 
edited sound ma y retain the nuance, timbre and ambiance of the 
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original recording. 

(4) System Overview 

(5) In a preferred embodiment, the invention is implemented for 
Macintosh computers running a Mac Operating System Version 7. 
However, the present invention is not limited to any particular 
hardware or operating system environment. Instead, those skilled in 
the art will find that the systems and methods of the present 
invention may be advantageously applied to a variety of systems, 
including IBM compatible personal computers running MS-DOS, Microsoft 
Windows or workstations running UNIX as well as specialized music 
keyboards and music workstation products. Therefore, the following 
description of specific systems are for purposes of illustration and 
not limitation. 

(5) FIG. 1 illustrates an example of a computer system used to 
execute the software of the present invention. FIG. 1 shows a 
computer system 1 which includes a monitor 3 with screen 5, cabinet 
7, keyboard 9, and mouse 11. Mouse 11 may have one or more buttons 
such as mouse button 13. Cabinet 7 houses a floppy disk drive 17, CD- 
ROM drive 19, and a hard drive (not shown) that may be utilized to 
store and retrieve digital audio information and software programs 
incorporating the present invention. Although a floppy disk 15 is 
shown as the removable media, other removable tangible media 
including optical disk and tape may be utilized. Cabinet 7 also 
houses familiar computer components (not shown) such as a processor, 
memory, and the like. So far this is a typical desktop computer 
system. 

(7) In order to be able to handle sound and music, computer system 
1 has a few extensions. Cabinet 7 houses Analog to Digital (A/D) and 
Digital to Analog (D/A) converters (not shown) . Those may be built 
into the computer system or a third party sound card may be added. 
Microphone 152 connects to the A/D converters and provides a 
representative source of audio information. The D/A converters 
connect to amplified speakers 162. A 'Ma;:Dl3iS£iiia^fcn^^ to a 
ser^at or other kind of I/O port of computer system 1 (I/O port not 
shown) . A MIDI device 180, typically a keyboard/synthesizer, connects 
to the MIDI interface. The connection may be bi-^drEiS¥ioEal; wJien_jone 
Pi^.YA^theJ^nD]^ , informatij:)n_^ 

tji^comput.er; the— c ompu^^-r-^Jg]^ turn^c^n^s end MIDI ^ odgs t^_jbii e 
s ynthesizer part of MI DI_dey ice 18 0. The sound ^ou tput of MIDI devic e 
180 conne£ts t^ is c^Ixe^ with the 

sound output of the D/A converters; optionally a juix er can be used 
(not shown) . 

(8) FIG. 2 shows a system block diagram of the computer system used 
to execute the software of the present invention. As in FIG. 1, the 
computer system includes monitor 3, keyboard 9, mouse 11, floppy disk 
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drive 11, and CD-ROM drive 19. The computer system further includes 
subsystems such as a central processor 102, system memory 104, I/O 
controller 106, display adapter 108, serial port 112, disk drive 116, 
network interface 118, analog to digital (A/D) converters 150, 
digital to analog (D/A) converters 160. Other extensions comprise 
microphone 152, amplified speakers 162, MIDI interface 170, and MIDI 
keyboard/synthesizer 180. Many of these subsystems are interconnected 
through system bus 122. Other computer systems suitable for use with 
the present invention may include additional or fewer subsystems. For 
example, another computer system could include more than one 
processor 102 (i.e., a multi-processor system) or memory cache. 

(9) Bi-directional arrows such as 122 represent the system bus 
architecture of the computer system. However, these arrows are 
illustrative of any interconnection scheme serving to link the 
subsystems. The computer system shown in FIG. 1 and FIG. 2 is but an 
example of a computer system suitable for use with the present 
invention. Other configurations of subsystems suitable for use with 
the present invention such as music workstations will be readily 
apparent to one of ordinary skill in the art. 

(10) In the preferred embodiment, A/D converters 150 can receive 
analog audio data from microphone 152 or other analog sound source, 
convert it to digital samples, and send those samples through bus 122 
to system memory 104, disk drive 116, or other interface subsystems. 
D/A converters 160 convert digital samples received from system 
memory 104, disk drive 116, or other interface subsystems via bus 
122, into analog sound data, then output the analog data to amplified 
speakers 162. MIDI interface 170 can (1) receive user input from the 
keyboard of MIDI device 180, and redirect the data through bus 122 to 
other sub-components of the system, and (2) receive data via bus 122 
and output MIDI data to the synthesizer part of MIDI device 180. The 
analog output of keyboard/synthesizer 180 is amplified and output by 
amplified speakers 162. 

(11) FIG. 3 shows a block diagram for an alternative system used to 
execute the software of the present invention. In this embodiment, a 
keyboard/synthesizer or "music workstation" product is used to embody 
the present invention. Audio enters processor unit 200 from sound 
input device 152 (e.g., a microphone) and is converted into digital 
samples by A/D converters 150. Analysis unit 202 (e.g., DSP) extracts 
musical parameters from the converted digital samples. Notes and 
other musical parameters are also entered in real time from user 
input device 180 (e.g., a MIDI keyboard/synthesizer) and/or user 
input device 182 (e.g., MIDI sliders). Those notes and other 
parameters represent the user's musical intention. 

(12) Controller unit 206 (e.g., a microprocessor) compares the 
parameters generated by analysis unit 202 with the ones entered 
through user input device 180 and/or user input device 182 in real 
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time. As a result of this comparison, time varying control functions 
for DSP algorithms are generated. The digitized samples from A/D 
converters 150 are also fed to processing unit 204 (e.g., DSP), which 
uses the time varying control functions generated by controller unit 
206 as control parameters. The processing occurs in real time while 
all other components of the system continue to work in parallel. The 
resulting digital samples are converted to an analog signal by D/A 
converters 160. The analog signal is fed to sound output device 162 
(e.g., an amplified speaker). The resulting sound retains many 
characteristics of the original sound like timbre, expression, and 
ambiance while some of its musical parameters (pitch, for instance) 
correspond to the parameters entered through input devices 180 and/or 
182 . 

(13) Processing unit 200 may be an independent unit housed in an 
enclosure with analog inputs and analog outputs as well as MIDI 
inputs and optionally MIDI outputs. Alternatively, processing unit 
200 may be a subcomponent of a musical workstation housing MIDI 
keyboard 180 and optionally microphone 152 and amplified speakers 162 
as well as processing unit 200 in one enclosure. 

(14) FIGS. 4a-4f illustrate various forms of musical 
representation. Musical parameters may be digitally stored in 
computer memory in many different ways. When presented to users, 
musical information takes one or more of various kinds of 
representations. FIG. 4a shows a short musical phrase in traditional 
music notation. 

(15) FIG. 4b shows the same phrase in piano roll representation — 
notes are represented as horizontal bars on a grid. Like traditional 
music notation, the horizontal axis represents time, while the 
vertical axis represents pitch. However, the length of a note is 
represented by the length of the bar while pitch is represented by 
the exact vertical position (no accidentals are used to alter the 
vertical position as in traditional music notation) . 

(16) The same phrase is shown in FIG. 4c in a list representation. 
Each line represents a note with the first three columns delineating 
the start time, the fourth, fifth and sixth representing the 
duration, while the last two columns represent the velocities of 
pressing and releasing the corresponding key on a musical keyboard. 

(17) FIG. 4d shows a textual musical representation suitable for 
editing with a word processor. The different kinds of note 
representation in FIGS. 4a-4d are shown for illustration purposes 
only. Although traditional music notation, piano roll, and list 
representation are commonly used, other ways of representing music 
information may be equally suitable for di^^ Pi#^vA^]^^ ^HM^^^ g 
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(18) FIG. 4e and 4f represent methods for viewing and editing 
continuous MIDI controller data. Controller events are displayed as 
single vertical lines, indicating their values, along a horizontal 
axis, representing their placement in time. In addition to basic 
controller data (volume, pan, pitch bend) , the Controller view may 
also be used for displaying and editing tempo events and key 
velocities for notes. 

(19) FIGS. 5a-5c show various representation of digital audio 
information. Digital samples are usually stored in memory or on hard 
disk drives as 16-bit two ' s-complement numbers. When presented to the 
user, digital sounds may take one or more of several kinds of 
representations . 

(20) FIG. 5a shows a waveform representation of a short digital 
audio fragment. The horizontal axis represents time while the 
vertical axis represents the amplitude of the samples. In this 
example the samples are very dense resulting in a c |a?iqfewi^ gp^ that 
cl'eaxTy^"]i^l^Row''TH'e e n e r g^^^f "^f fl@^^SlS^^chang!©s in time. In some 
cases like the one in FIG. 5a, we can clearly see the shape of 
individual notes. However, it is nearly impossible to distinguish the 
pitch of the sound. 

(21) FIG. 5b shows a 3-D spectral representation of the sound. The 
sound is cut into small time slices (buffers) , each slice is analyzed 

(usually using the Fourier transform) , and an instantaneous spectral 
representation of that slice is generated. The x-axis represents 
freguency while the y-axis represents amplitude. Slices are arranged 
one after the other on the z-axis that represents time. This 3-D 
spectral representation allows us to see the evolution in time of 
each spectral component. 

(22) A different method of showing the evolution in time of the 
different spectral components, also known as partials or harmonics, 
is the sonogram, shown in FIG. 5c. The horizontal axis represents 
time, the vertical axis represents frequency. The amplitude of each 
spectral component is shown by the intensity of the color or gray, 
the darker the color the higher the amplitude. For black and white 
displays the width of the lines representing spectral components may 
show the amplitude. 

(23) As with music representation, the different ways of displaying 
digital audio information shown in FIGS. 5a-5c are for illustration 
purposes only. Other kinds of representation may be equally suitable 
for displaying and editing digital audio. 

(24) Definitions 

(25) Terminology for this patent application is defined below. 
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(26) Attack: An attack is the sound of the beginning of a note. 
During the attack, which is often the loudest portion of a note, 
there are often brief percussive sounds that are not present later in 
the note. Attack sounds contribute greatly to an instrument's 
distinguishing characteristic . 

(27) Continuous controller: Some controllers generate signals that 
have a limited number of states (e.g., a foot switch). Continuous 
controllers, by contrast, generate a smooth, continuously varying 
function, to control sound characteristics such as volume, pan, or 
pitch bend. 

(28) Controller: A controller is a physical MIDI device that sends 
MIDI data to control a performance. Additionally, the term controller 
refers to the MIDI signal corresponding to a physical controller. 
Knobs and levers on keyboards, foot pedals and other devices send 
signals that can, for example, change the pitch of sound, introduce 
fluctuations in sound (called vibrato and tremolo) , vary volume, or 
sustain notes longer than their normal length. 

(29) Controller view: This editing window provides an excellent 
method of viewing and editing continuous MIDI controller events. 
Controller events are displayed as single vertical lines, indicating 
their values, along a horizontal axis, representing their placement 
in time. Controller views for volume and pitch bend are illustrated 
in FIGS. 4e and 4f. 

(30) DSP: Digital signal processing (DSP) is the processing of a 
range of samples in a digital audio waveform. Common -DS-P^-lge^tthms 
a'Sa^o^r^G^ha^ig errr,e v.er^b er a:tl^^ 

the like. DSP algorithms can be executed on a computer's main 
processor or on a special DSP co-processor. 

(31) DSP module: A DSP module is a subprogram for performing a 
specific DSP function, such as pitch-shifting or time expansion or 
compression . 

(32) Duration: Duration is the length in time of a note. On an 
organ, for example, a note's duration is determined by how long its 
key is held down. A quarter note has twice the duration of an eighth 
note . 

(33) Dynamics: Dynamics are the variations in loudness of a passage 
of music. 

(34) Expression: Expression includes the variations in dynamics, 
timing and pitch that convey nuance and feeling in music. 

(35) Floating point samples : ^^ga$c:ggg^pS Hitei§CT;^amples 
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consist of 16 bit integers. Floating point samples are represented 
with floating point numbers that consist of a group of digits and an 
exponent that determines where the decimal point is placed. The 
number 2.3459 E9 is a floating point number equivalent to 
2, 345, 900; 000, where the number following the "E" is the exponent. 
Floating point samples can represent a much wider dynamic range of 
sound volumes than integer samples. 

(36) List window and list view: One format for representing a 
musical performance consists of a list of notes in the order in which 
they are to be played. Musical parameters for each note are included 
in each list entry, such as the bar, beat and unit when the note is 
to begin, and its pitch, velocity and duration. Other formats include 
piano roll and standard musical notation. 7\n example of list 
window/list view is illustrated in FIG. 4c. 

(37) MusieaJr^^^iTrsJix : MIDI is a 
communications protocol that permits a wide variety of electronic 
music equ ipment (e.g . , keyb oar4g7 7~el-'ect-rQnl-c-^d-rumsx Zg^^ 
comput'exsIIand~recQ-rdXng--equTpm^ ) to communicate with each other. 
MIDI data includes performance data as opposed to sound or waveform 
data. It specifies which note (pitch) to play, when notes begin and 
end, how loud notes are to be played and the electronic instrument on 
which notes are to be played. MIDI does not describe the detailed 
sound of instruments. 

(38) Husl7ca3;ij^^ A musical performance can be represented 
as a collection of time-ordered notes, and the notes themselves 
represented with a set of numeric characteristics, known as musical 
parameters. Pitch, duration, attack and other "envelope" parameters, 
along with spectral content, are musical parameters. 

(39) Notation: Notation is the conventional way in which music is 
represented consisting of staves, key and time signatures, with notes 
represented as solid or empty heads with tails that have flags 
representing their lengths. Notation also uses special symbols for 
rests (periods of silence) and dynamics. An example of music notation 
is illustrated in FIG. 4a. 

(40) Piano roll: A piano roll display is a graphic representation 
of a region of music in which the Y-axis represents the pitch of a 
note and the X-axis represents time. The length of a note in piano 
roll format represents its duration. The format takes its name for 
the perforated paper rolls used in player pianos. An example of piano 
roll is illustrated in FIG. 4b. 

(41) Pitch: Pitch is the fundamental frequency of a note. As notes 
are played in an ascending scale, from left to right on a music 
keyboard, the pitch of each note is higher than the next. In addition 
to the fundamental frequency, notes contain higher frequency 
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overtones that combine to give an instrument its characteristic 
timbre . 

(42) Pitch bend: Pitch bend is a continuous MIDI controller message 
that usually instructs an instrument to raise or lower the normal 
pitch of a note. 

(43) El^a^yii&t : ^^ pl^^ip ijE ^ audio by computer software is 
often done with playlists. With a playlist, original digital audio 
source material, stored as a collection of files, may be edited into 

a single composition without changing the content of the original ' 
files, Playlists are comprised of audio events, playing one after ' 
another and/or concurrently; audio events point to a specific region 
(with start and end points) of a particular source audio file. Audio 
events in a playlist may a lso cont ain volume and pan information to 



(44) Polyphony: Music in which two or more notes are playing at the 
same time is polyphonic. Sequences of notes to be played concurrently 
on separate instruments are called voices. 



(45) <^^m^^^^SW^ In an actual musical performance by a human 
musician, there are variations in note timing, volume and pitch that 
may deviate from the intention of the composer. Quantization compares 
electronic codes generated from a human performance with an ideal 
performance and adjusts the coded values partially or completely in 
the direction of the ideal. Playing back the adjusted codes results 
in a more precise performance. 

(46) Spectral content: The content of a complex musical note may be 
represented as the sum of a series of simple sine waves of different 
frequencies, phases and amplitudes. The lowest, fundamental frequency 
is often the loudest and is the note's pitch. The unique blend of 
fundamental and higher frequencies, referred to as the note's 
spectral content, gives an instrument its distinguishing timbral 
characteristics. For example, "brightness" is caused by high 
frequency spectral content. The process of equalization modifies the 
overall spectral content of a passage of music, 

(47) Tempo map: A Tempo map is a representation of how the pace of 
a passage of music varies with time, 

(48) Text form: Another format for representing a musical 
performance consists of a list of musical parameters for each note 
(i.e., bar, beat, actual note, and note duration) in the order in 
which they are to be played. An example of text form is illustrated 
in FIG. 4d. 

(49) "Wl^s&m^^ When a note is pilygd~-fevn^ar^-SiS^ the 
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strength with which the key is struck is measured. When the note is 
then played, this measurement, known as velocity, can be used to 
control the note's initial loudness or other parameters. 

(50) Waveform: A waveform is a representation of an analog audio 
signal in which the Y-axis represents instantaneous amplitude and the 
X-axis represents time. An example of a waveform is illustrated in 
FIG. 5a. 

(51) Basic Principles of the Invention 

(52) ^55G^6^is a block diagram showing the basic principle of this 
invention. Original Digital Audio material 300 will be processed by 
DSP Modules 314, 316, and 318 resulting in New Digital Audio material 
320. Certain music-like aspects of Original Digital Audio 300 will be 
modified while other parameters will stay the same. Some of the 
music-like parameters of Original Digital Audio 300 are coded and 
digitally stored as Original Musical Information 302 . <MlI)i^s one of 
many possible representation for this information. The New Musical 
Information 304 (again MIDI is one of many possible representations 
for this information) corresponds to the intended music-like 
parameters of New Digital Audio 320 after the modification. 

(53) We are attempting the transformation of the music-like 
parameters of "Original Digital Audio" 300, which are encoded by 
"Original Musical Information" 302 into new music-like parameters 
encoded by "New Musical Information" 304. We use Comparator 306 to 
find the differences between Original Musical Information 302 and New 
Musical Information 304, this operation being possible when Original 
Musical Information 302 and New Musical Information 304 have a 
structure suitable for comparison and encode similar music-like 
parameters. The output of Comparator 30 6 is the set of time varying 
control functions 308, 310, and 312. Any number of time varying 
control functions can be generated by comparator 306 depending on the 
number of similar music-like parameters encoded by Original Musical 
Information 302 and New Musical Information 304 as well as user 
preferences . 

(54) FIG. 6 suggests Original Musical Information 302 and New 
Musical Information 304 are encoded as MIDI and presented to the user 
in piano-roll and strip chart form. These are commonly used encoding 
and representation methods, however, many other encoding techniques 
and means of graphic representation may be used. FIG. 6 shows 
Original Musical Information 302 and New Musical Information 304 
representing notes and wljome^oharigas . Notes are shown in piano roll 
form with pitch and duration as visible parameters while volume 
changes are represented as continuous controller events in strip 
chart form. Comparing the pitch of each note from Original Musical 
Information 302 with the pitch of eacii note from N'ew-=Mn"sg.^M 
Information 304 results in Time Varying Control Function 308. 
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Similarly, GQ mpAri ng^jdur aj^^^ s figgjnr-Qrg;Lg.i'^ Information 302 

with durations f rom ^^^__M]j^i^^ results in Time 

Varying Control Function 310. The differences in volume changes from 
Original Musical Information 302 and New Musical Information 304 
result in Time Varying Control Function 312. 

(55) Q^i^Snwte^igri-ta^ is processed by a set of Digital 
Audio Processing (DSP) Modules. FIG. 6 suggests three DSP Modules: 
314, 316, and 318. However, any number of DSP Modules may be used 
depending on the number of time varying control functions generated 
by comparator 306 and user preferences. FIG. 6 suggests that the DSP 
Modules 314, 316, and 318 are connected in series, yet other 
configurations may be used. 

(56) Pitch Shifting DSP Module 314 processes Original Digital Audio 
300 according to Time Varying Control Function 308. Pitches are 
raised', lowered or left unchanged in such a way that the pitches of 
the resulting output correspond now to the pitches encoded by New 
Musical Information 304. Time Stretching DSP Module 316 processes the 
output of Pitch Shifting DSP Module 314 according to Time Varying 
Control Function 310. Notes are lengthened, shortened, or left 
unchanged in such a way that the durations of the resulting output 
correspond to the duration encoded by New Musical Information 304. 
Similarly, Gain DSP Module 318 processes the output of Time 
Stretching DSP Module 316 according to Time Varying Control Function 
312 the resulting output having volume changes corresponding to the 
volume changes encoded by New Musical Information 304. 

(57) The output of Gain DSP Module 318 is New Digital Audio 320 
that contains all the transformations performed by DSP Modules 314, 
316, and 318. The pitches, durations and volume changes of New 
Digital Audio 320 correspond now to the pitches, durations and volume 
changes encoded in New Musical Information 304. All other music-like 
and sonic parameters such as timbre, expression, and ambiance remain 
the same as in Original Digital Audio 300. Thus music-like parameters 
from our source material Original Digital Audio 300 have been 
modified according to the differences between Original Musical 
Information 302 and New Musical Information 304, resulting in new 
material New Digital Audio 320. 

(58) FIGS. 7, 8, 9, and 10 further illustrate the relationship 
between the Original Digital Audio (300 in FIG. 6), the Original 
Musical Information (302 in FIG. 6), the New Musical Information (304 
in FIG. 6), the Time Varying Control Functions (308, 310, and 312 in 
FIG. 6), and the New Digital Audio (320 in FIG. 6). In FIG. 7 the 
Original Music Notation 301 and the Original Piano Roll 302 are two 
different methods of representation for the same Original Musical 
Information. 

(59) Underneath the Original Piano Roll 302 and aligned to it we 
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see the Original Digital Audio 300 displayed as waveform. Original 
Digital Audio 300 has several music-like characteristics, note 
pitches and note durations being two of them. Those two attributes 
are coded in the Original Musical Information and displayed as 
Original Music Notation 301 and Original Piano Roll 302. Pitches are 
hard to identify in waveform display but individual notes may be 
apparent in some cases like the one we are examining here. The arrows 
305 show the correspondence between the beginning of the second note, 
the fourth note, the sixth note, and so on, in the Original Piano 
Roll 302 and in the waveform display of Original Digital Audio 300. 

(60) In FIG. 8 the Original Musical Information has been replaced 
by New Musical Information represented as New Music Notation 303 and 
New Piano Roll 304. Since pitches are hard to identify in waveform 
display we are going to show only changes in the durations of the 
notes. We can see that the rhythm has been changed by ciompSSiSg' 
Qrigi^l'I^Siu^^ with N.e.W'~MuslXZN.ata:tlo^ or Original 
Piano Roll 302 with cN^e^ZPi^^^o^^^U'^^ Moreover the arrows 305 show 
that the beginning of the second note, the fourth note, the sixth 
note, and so on, in the waveform display of Original Digital Audio 
300 and in the New Piano Roll 304 no longer correspond. 

(61) FIG. 9 shows the process of comparing Original Musical 
Information with New Musical Information in order to generate Time 
Varying Control Functions. In this example we are comparing only the 
durations from the Original Musical Information (represented in piano 
roll form as Original Durations 302) with the durations from the New 
Musical Information (represented in piano roll form as New Durations 
304) . The result is Time Varying Function 310 suitable to control a 
Time Stretching DSP Module (not shown) . 

(62) The horizontal axis for Time Varying Control Function 310 
represents the time of the Original Digital Audio (not shown in FIG. 
9) . The vertical axis represents the amount of time stretching; an 
amount smaller than 1 means time compression, an amount greater than 
1 means time expansion, and an amount equal to 1 represents no 
change. As we can see the first, third, and fifth notes will be 
expanded to 1.5 times their original durations; the quarter notes 
become dotted quarter notes. The second, fourth, and sixth notes will 
be compressed to half their original durations; the quarter notes 
become eighth notes. The duration of the seventh note will not 
change. Similarly notes 8, 10, and 12 will be expanded, notes 9, 11, 
and 13 will be compressed, while note 14 will be left untouched. 

(63) The result of processing the Original Digital Audio 300 from 
FIG. 7 and FIG. 8 with a Time Stretching DSP Module (not shown) 
controlled by Time-Varying Control Function 310 from FIG. 9 is shown 
as New Digital Audio 320 in FIG. 10. New Music Notation 303 and New 
Piano Roll 304 are also shown. The arrows 305 accentuate the 
correspondence between the beginning of the second note, the fourth 
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note, the sixth note, and so on, of the New Piano Roll 304 
representation of the New Musical Information and the waveform 
representation of New Digital Audio 320. 

(64) We should ei^^ai^e how easy it is for a musician to 
understand the differences between the Original Musical Information 
and the New Musical Information as represented in either Traditional 
Music Notation or Piano Roll form. By contrast, we should note how 
unintuitive the Time-Varying Control Function 310 from FIG. 9 is. By 
letting the user express their desired changes in a familiar mode of 
representation and then electronically generating a time-varying 
function and supplying it as control input for an appropriate DSP 
Module, we provide for a much friendlier and more effective 
environment for the editing— aSHiit:r:an:s:f;o:rm^^ of recorded digital | 
audio material. 

(65) Digital Audio Format 

(66) The^i^xtad— a-udxo^==u^^ and as destination in the 
present invention (Original Digital Audio 300 and New Digital Audio 
320 in FIG. 6) may be of any sample rate and any sample size. In 
professional settings, 16-bit two ' s-complement linear samples at 44.1 
kHz or 4 8 k Hz are commonly used. The d±g:i:tajr;^a^u^A^ 

ha^rd=::dijsk, ^RAM^maga-e-tG-^op^X , CDER~QM7IICD:^Auai37IIoZZtR^ 

any random access type of media may be used, even commercial CDs 

e^l^~beI5iaea3:alThongh~^ 

(67) If the dl-gri-=fca-l^a:udd-Q=i~srrs'go^^^ or any 
other kind of linear media, time compression and expansion are hard 
to achieve. However, pitch shifting, gain change, filtering, and any 
kind of time invariant processing may equally be applied to digital 
audio on tape. Time compression and expansion are still possible if 
thB-^,ap:er=d-e^elc'^^ 

(68) When the digital audio is stored on hard disk or other non- 
volatile storage medium, any kind of file format may be used. For 
example, raw data files containing just the samples are usable. AIFF 
and Sound Designer II are popular sound file formats for the Apple 
Macintosh computers, while on Microsoft Windows, WAV files are widely 
used. All these and other file formats may be use to store the 
Original Digital Audio (300 in FIG. 6) and/or the New Digital Audio 
(320 in FIGv-- 6^' . Data compression is also viable as long as either 
(1) the DSP Modules can process compressed digital audio or (2) a 
decoder module is used to decompress the stored data before 
processing. In the second case an encoder module may be used to 
compres^ the result of processing. MPES-^AudiOvzrr-DoibyiSz^^ and AD^ 
PCM— a^-^gtiisromerof the many commonly used data compression methods 
applicable to this invention. 

(69) Floating point samples are also suitable if the DSP has 
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floating point capabilities or if the data is converted before and/or 
after processing. Other encoding methods like .mu.-Law or a-Law may 
also be used. 

(70) Finally, analog audio may also be utilized. Analog audio may 
be digitized on the fly, and the rest of the processing will be 
identical to that of digital audio. Because the audio is not stored 
on disk, the same restrictions as those for digital tape apply. 
Moreover, the DSP Modules may be replaced by digitally controlled 
analog processing modules and then both the source material as well 
as the resulting new program can be analog audio. In the case where 
the analog processing modules are controlled by analog signals the 
digital time-varying control functions may be converted to analog 
signals before being fed to the processing modules. 

(71) Digital Audio Representation 

(72) Both Original Digital Audio (300 in FIG. 6) and New Digital 
Audio (320 in FIG. 6) may optionally have a visual representation. 
The most common display method for digital audio is the waveform (see 
FIG. 5a) . Other options may include 3D spectral display (see FIG. 5b) 
and sonogram (see FIG. 5c) as well as any other method of graphical 
representation of digital audio. 

(73) Music-like and Other Audio Parameters 

(74) The GoTf€entr:o#r:t-he^ may be music with 
such distinguishable elements and characteristics as notes, pitches, 
durations, rhythm, tempo, dynamics, accents, and the like. Sources 
other than music may be used especially when they exhibit music-like 
characteristics. For instance, speech may have cadence and 
intonation, while sound effects may have rhythm or dynamics. Any 
audio source with parameters that may be coded, modified, .^ompared., 
and the differences used as time-varying parameters to control 
processing functions are suitable for use with this invention. 

(75) Some of the characteristics of the audio material may belong 
to common musical practice and may be represented by traditional 
musical notation. More generally, a parameter may be any kind of 
quantifiable sonic characteristic of the audio. 

(76) One important category is discrete parameters such as pitches 
(note names) and durations. Other characteristics such as loudness, 
tempo changes, or expression can also be expressed through discrete 
parameters such as dynamics markers (e.g., forte, piano, crescendo), 
tempo changes markers (e.g., accellerando, ritardando) , or others 
such as accents (e.g., sforzando) . 

(77) Some parameters may be viewed as continuous time varying 



127 0 0 1 :4343 C: APPS ea cac e ea 200307241 15016541 m e 



C ie%20Ne I 7 24 03 



DOCUMENT-IDENTrFIER: US 5792971 A 



Page 21 of 47 



functions. The way the fundamental pitch (or a series of fundamental 
pitches) varies in time may be represented by a continuous function. 
Loudness, brightness, rate and amount of tremolo, /r^aafei^^^aStgiaount of 
vibrato, amount of direct versus reverberated sound, ^^^B^^oSgfebe n , as 
well as many other characteristics of the sound may be represented by 
continuous time varying functions. Continuous functions are also 
suitable for the representation of the characteristics of spoken 
words and other non-musical voice recordings. 

(78) eiS^sX^GaV-arid^^^^ 

(79) Discrete parameters may be coded in many different ways and 
stored in RAM or on permanent computer storage devices. Continuous 
time varying function may be represented and stored as sampled 
points, as connected line segments, or any other kind of 
approximation. Several languages and standards are available for 
digitally representing musical and sonic parameters. Most of those 
languages are suitable to encode the Original Musical Information 

(302 in FIG. 6) and the New Musical Information (304 in FIG. 6) . 

(80) One of the most common encoding method is the MIDI standard. 
MIDI allows notes to be represented with pitch (note number) , 
durations (interval between note "on" and note "off", or duration in 
MIDI files), and strength of the attack (velocity). Continuous pitch 
variations can be represented as Pitch Bend relative to the note's 
pitch. Continuous loudness variations can be represented as Volume 
controllers. Continuous brightness variations can be represented as 
any other MIDI controller. Other quantifiable parameters may be also 
represented as MIDI controllers. Other languages and standards may be 
suitable for use with this invention, but due to its prevalence and 
acceptance MIDI is the preferred choice. 

(81) Musical and Audio Parameters Representation 

(82) Both Original Musical Information (302 in FIG. 6) and New 
Musical Information (304 in FIG. 6) may optionally have a visual 
representation. The most common coding method for musical information 
in MIDI. Some of the most common ways of displaying MIDI are 
Traditional Music Notation (see FIG. 4a), Piano Roll (see FIG. 4b), 
and List (see FIG. 4c) . 

(83) Time Varying Control Functions 

(84) The nature and number of Time Varying Control Functions (308, 
310, and 312 in FIG. 6) depends on (1) the nature and number of 
similar parameters encoded into the Original Musical Information (302 
in FIG. 6) and the New Musical Information (304 in FIG. 6) and (2) 
the number and kind of DSP Modules (314, 316, and 318 in FIG. 6) . 
Time Varying Control Function may be represented and stored as 
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sampled points, as connected line segments, or other kinds of 
mathematical description . 

(85) One of the goals of this invention is to provide musicians 
with familiar and intuitive tools. One way to achieve this goal is to 
insulate them from what may be perceived as complicated technical 
data and concepts such as Time Varying Control Functions, However, in 
some cases users may be given the option of graphically viewing and 
even editing the Time Varying Control Functions, 

(86) Not all musical parameters have a one-to-one correspondence 
with a Time Varying Control Function, In some cases comparing several 
sets of musical parameters may result in only one control function. 
In others, the differences between one pair of similar parameters may 
generate more than one control function. 

(87) For instance, comparing the duration of each note from 
Original Musical Information (302 in FIG, 6) with each note from New 
Musical Information (304 in FIG. 6) as well as comparing the duration 
of each silence from both sets of musical information, will result in 
a Time Varying Control Function suitable for controlling a Time 
Stretching DSP Module (see FIG. 9) . Comparing tempo changes between 
the two sets of musical codes will also result in a Time Varying 
Control Function suitable for controlling a Time Stretching DSP 
Module. It is desirable to combine the two functions (by multiplying 
one with the other) in order to provided one single Time Varying 
Control Function. In this way we minimize the CPU requirements; 
multiplying two functions is less demanding than using two separate 
Time Stretching DSP Modules, one for duration changes, the other one 
for tempo changes. 

(88) Another similar example is the use of MIDI note numbers 
(discrete pitches) and pitch bend events (small continuous variations 
around each note) to code pitch variations. Comparing MIDI note 
numbers from Original Musical Information (302 in FIG. 6) with MIDI 
note numbers from New Musical Information (304 in FIG. 6) will result 
in one Time Varying Control Function suitable for controlling a Pitch 
Shifting DSP Module. Comparing pitch bend events from the two sets of 
codes will result in a similar Time Varying Control Function. The two 
may be combined into a single control function in order to minimize 
the CPU requirements. Thus the Comparator (306 in FIG, 6) may receive 
both note numbers and pitch bend events as input while its output may 
consist of a single control function. 

(89) Similarly, both the changes in the velocity of MIDI notes, a 
discrete parameter, and volume controller changes, a continuous 
parameter, represent variations in loudness. The Comparator may 
combine the two and output a single Time Varying Control Function 
suitable for controlling a Gain DSP Module. Furthermore, the amount 
and rate of tremolo may additionally affect loudness while the amount 
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and rate of vibrato may add to the pitch variations. 

(90) On the other hand we may have one set of parameters generating 
several Time Varying Control Functions. For instance, the velo^city of 
each MIDI note represents how fast (i.e. how hard) a key of a MIDI 
keyboard has been pressed. Velocity is first of all a gestural 
parameter that may translated into one or more sonic parameters in 
several ways. One commonly used technique is to use velocity to 
control both the loudness and the brightness of a note; the harder 
you hit a key the louder and brighter the resulting note. Thus 
comparing the velocity of each MIDI note from Original Musical 
Information (302 in FIG. 6) with the velocity of each MIDI note from 
New Musical Information (304 in FIG. 6) may result in two control 
functions, one suitable to control a Gain DSP Module, the other 
suitable to control a Filter DSP Module. 

(91) T r:an£l^^^^|^rs »ig@faia^^ 

(92) In most cases changes in musical information translate easily 
into changes suitable to control a DSP Module. For instance, duration 
changes may be expressed as a ratio between the new duration and the 
old one (see FIG. 9) . Thus a new duration that is twice as long as an 
old duration will be expressed as a ratio of 2 while a new duration 
that is half the duration of an old duration will be expressed as a 
ratio of 1/2 or 0.5. These ratios may be used directly to control a 
Time Stretching DSP Module. 

(93) Tempo changes are the inverse of duration changes. For 
instance, at a' tempo of 60 beats per minute a quarter note is 1 
second long; at a tempo of 120 beats per minute that is twice as 
fast, a quarter note is half a second long. Thus the time stretching 
ratio is the inverse of the tempo ratio. In order to control a Time 
Stretching DSP Module we need to take the old tempo and divide it by 
the new tempo, as opposed to dividing the new duration by the old 
duration. A time stretching ratio of 1 represents no change, a time 
stretching ratio greater than 1 represents time expansion, while a 
time stretching ratio less than 1 represents time compression. To 
combine two time stretching ratios we multiply the two fractions. 
Thus multiplying two or more time stretching control functions 
results in the combination of those functions. 

(94) Loudness changes may be exposed in dB or as a gain multiplier. 
The dB scale is logarithmic, thus a change of 0 dB means no change, a 
positive change means an increase in loudness, while a negative 
change represents a decrease in loudness. Combining two or more 
loudness functions expressed in dB is perform by adding the functions 
together. On the other hand a gain multiplier of 1 represents no 
change, a gain multiplier greater than 1 means an increase in 
loudness, while a gain multiplier less than 1 represents a decrease 
in loudness. Multiplying two or more gain control functions results 
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in the COTibirBiailronll^ f those functions. To translate between loudness 
changes, gain multiplier changes, and vice-versa we use the formulas: 

(1) L=20 log (g) 

(2) g=10,sup.L/20 

(95) where L is the loudness change in dB and g is the gain 
multiplier. 

(96) ^P irfech— c_h agjg glg^^ s ed as musical intervals or as a 
frequency ratio. Musical intervals may be expressed as a number of 
semitones or in cents (one semitones has 100 cents) . A positive 
number of semitones means shifting the pitch up for that number of 
semitones, a negative number means shifting the pitch down, while 
zero means no change. Combining pitch shifting functions expressed in 
musical intervals amounts to adding the functions. On the other hand 
a frequency ratio of 1 means no change, a frequency ratio greater 
than 1 means pitch shifting up, while a frequency ratio less than 1 
means pitch shifting down. Multiplying two or more frequency ratio 
functions results in the combination of those functions. To translate 
between musical intervals, frequency ratios, and vice-versa we use 
the formulas: 

(3) s=12 log. sub. 2 (f) 

(4) f=2 .sup.s/12 

(97) where s is the number of semitones and f is the frequency 
ratio . 

(98) DSP Modules 

(99) The structure and programming of DSP modules are dependent on 
the musical characteristics that are to be changed. To change the 
duration of timing of notes, rests, as well as tempo changes, a time 
searching module is employed. This module is able to compress or 
expand the timing without changing pitch. A pitch shifting DSP module 
can change the pitch of the subject without changing the timing. A 
gain module is used to change volume and dynamics. Filters may be 
used to alter the spectral content. 

(100) Automatic Extraction of Musical Parameters 

(101) In the basic process of the invention (FIG. 6), original 
musical information is entered manually. It can be very difficult to 
enter the desired nuances of a musical performance, therefore the 
computer may be used to extract this data from digital audio in a 
very precise mode. Hence, an enhancement to the invention is added. 
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(102) As illustrated in FIG, 11, the Analysis Module 322 extracts 
musical information from original digital audio 300. Extracted 
musical information may be encoded in the form of MIDI data, 
including notes and continuous controller messages. 

(103) MIDI affords an intuitive and familiar environment for 
displaying and editing the extracted musical information: pitches and 
durations for individual notes with Notation and Piano Roll windows 

(FIGS. 4a and 4b), continuous controller data (such as volume, 
brightness and pitch bend) in Controller and List views (FIGS. 4c, 
4e, and 4f ) . 

(104) Optionally, an Analysis Guide 324 may be used to control 
certain functions of the Analysis Module. For instance, the computer 
can generate continuous controller information in a very precise 
manner (better than the humans) ; however, it is easier for a human to 
decide whether a change in pitch, for example, should be treated as 
one note with much pitch bend or as several notes with less pitch 
bend. The Analysis Guide helps making such decisions. Thg2^g^a.y_-af4^n 
b e-^a^r ep eaf^=^p-r 0i5es^s="0'^^ 



note~s7~-mo-re— an-ai=y-sis:;==^mo-re^^ 

(105) The rest of the procedure is the same as the basic process 
(FIG. 6) . Extracted musical information 302 is_an important reference 



f orjt rans form ing the or.i:g:inS=^i-grtai-^audl-o-^s'o^ 
CO dies2]3W7^^]wte 1 
in^^o'rma rran'^3'0'2'; provi^^theznecessaxy^i-n-s-tr_uc_^^^ 
di^i=tS==iLwil=o^3-2^0^^ ^-"^'^'^''^ 

(106) Editing Musical Information 

(107) The basic process (FIG. 6) uses new musical information 
entered manually. Sometimes it is difficult to enter complicated 
continuous data. 3 a:^rs~eag jreg-^tg— t the original information, 
especially when generated automatically from the Analysis M odule (322 
in FIG. 11). FIG. 12 illustrates how the original^inf ormatTorClSL is 
transformed through the fEgirtxng^;^^ resulting in new musical 
information 304. 

(108) gditln g^Mo dulre— 3-2'6? imposes changes on original musical 
information 302. These changes, representing new musical information 
304^axe^cgmpared^^^^^^ information 302 and the 
^^f'fe-rences/are then used as control input for processing by the 
appropriate DSP Module (314, 316, 318) . 

(109) Once the musical information has been extracted into the 
domain of MIDI, there are numerous . editing possibilities. 

(110) Pitches of individual notes or groups of notes are easily 
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changed by manually dragging them (with the mouse) to new pitches in 
the Notation and Piano Roll (FIGS. 4a and 4b) windows. Flat and sharp 
pitches may be fixed by editing pitch bend data in the Controller 
view (FIG. 4f ) ; articulations, often expressed as pitch fluctuations, 
may also be addressed with the appropriate drawing/editing tool in 
the Controller View (FIG. 4f) . And, of course, entire phrases may be 
transposed-diatonically, modally, or with custom transposition maps. 
Changes in pitch are processed by the Pitch Shifting Module 314. 

(Ill) rnidl'^ri"ciirari""^ lengths may also be changed 

manually in the Notation and ®i"Sno Roll (FIG. 4a and 4b) windows. 
Rhythmic placement of notes or phrases may be time corrected using a 
feature called Quantize. Tempo for original musical information is 
quite easily changed by inserting new tempo events in either List or 
Contr.O-lJLer views; in fact, c;rea-ti-ngl,acce.l:exan^^ 



qurte easy in tK¥^euntroi"ie'r==v-i=ew^Wlth the appropriate drawing tools. 
Changes with regards to time are processed by the Time Stretching 
Module 316. 

(112) V-o4=^ume's=^o^f^i^dl=^vl^-u^^^ notes may be altered by editing their 
respective velocity values. Dynamics for groups of notes or phrases 
may be changed or scaled by drawing volume events (continuous 
controller 7) in the Controller view (FIG. 4e) ; in fact, producing 
crescendos and decrescendos is quite easy in the Controller view 
(FIG. 4e) with the appropriate drawing tools. Changes in amplitude 
are processed by the Gain Module 318. 

(113) The primary strength of this aspect of the invention is to 
offer new editing possibilities for digital audio. Although these 
editing capabilities are not new in and of themselves (they are 
commonly found in most MIDI sequencing software), they are however 
new to the application of digital audio editing. 

(114) Analysis and Editing (Studio Vision Pro 3.0) 

(115) The preferred embodiment of the invention is exemplified in 
Opcode's Studio Vision Pro (FIG. 13), and may be broken down into 
four phases: (1) Musical information is extracted from the original 

(monophonic) audio source and encoded in the form of MIDI data, (2) 
The desired edits are performed on the extracted musical information 
and is transformed into new musical information, (3) The new musical 
information is compared to the original musical information and the 
appropriate DSP control parameters are generated, (4) The differences 
between the original and new musical information provide the 
necessary control codes to transform the original audio into a new 
one . 

(116) Phase One 
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(117) A monophonic digital audio file 300 is analyzed and its 
musical characteristics are extracted providing original musical 
information 302. For instance, a recording of a flute passage 300 is 
transformed into MIDI data 302 with individual notes indicating the 
basic melody, pitch-bend data reflecting the subtle articulations, 
and note velocities and volume data the phrasing and dynamics. 
Optionally, the extraction of musical information 302 may be assisted 
by the Analysis Guide 324, 

(118) Once these encoded control parameters are in the MIDI domain, 
they are intuitively displayed in the appropriate editing windows: 
notes in the Notation and Piano Roll windows (FIGS. 4a and 4b), 
continuous controller data in the Controller and List views (FIGS. 4e 
and 4f ) . 

(119) Phase Two 

(120) Altering a variety of aspects of the original musical 
information 302 is quite easy in the MIDI domain, which represents 
exciting possibilities for transforming recorded digital audio. 

(121) If the original flute recording 300 contains some flat or 
sharp notes, they can be smoothed out by editing or altering pitch 
bend data; if the key of the original performance is wrong, it can be 
transposed. If the dynamics of a particular section is not quite 
right, volume data can be added or changed to achieve the desired 
balance; and if tempo is determined to be too slow or fast, new 
tempos can be inserted. 

(122) Additionally, expressive aspects of the original performance 
300 can be altered or enhanced. Phrasing can be changed from legato 
to staccato by editing individual note lengths; timing can become 
more strict or loose by using time correction routines (quantize) ; 
and, subtle crescendos and decrescendos can be added by ramping 
individual note velocities. 

(123) Phase Three 

(12 4) ^ ^^^ ^ ^W^ tl^^^^^^^MSMlMmmm^ ya,€^B5 ^i,e^w^^^ 

i n#o rma t"ri 304 i s ^^^^aaL^r^ed^^dfeerols^ info rma t i o n 30 2 . 

The comparator 306 analyzes both sets of information and figures out 
their differences. Because the original musical information 302 is a 
direct link to characteristics of the original source audio, {the 

GOnt'rp l-^GOd 

ne,ee:ssax.y_:t±m'§-v:a"^^^^^ 

Ti^w:3aifaii3i:HgM^ ~~ 

(125) Phrase Four 
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(126) OgrCglZfEellcorEteQ the appropriate DSP modules 314, 
316, 318 are generated, cg~n:gwijai:gxta^ri^a^ 

p^oces;S3jigiit:lxez:o:^^^^^^ CSaSg^e^s^iliZpl"^^^^ 

the Pitch Shifting module 314; changes in note lengths and tempo are 
processed by the Time Stretching module 316; and, changes in note 
velocities and volume data are processed by the Gain module 318. 

(127) An Cii^o^tan^Ciraspqjst of the invention is that a ffe^==auSI^=?-i5*g^ 

rlgiQiri'sZgregrfee^^^ 

.02^ cieieted. Thus tfie technology represents "constructive" (not 
destructive) editing where the original material 300 is not lost. 
Optionally, the ori ginal audio 300 _may be replaced by the 'Sl^^izaudjip 
3 2 J l^Lf — de sj rr e du Ct o—s ave—har d~drs"k"~spa~c e7 — f or— iS^tanc e ) . 

(128) Using an Audio Guide to Generate New Musical Information 

(129) FIG, 14 illustrates a variation on the source for the new 
musical information 304 to which the original musical information 302 
is compared. In previous examples, the new musical information 304 is 
derived from edited or existing MIDI data. In this variation of the 
invention, the source of the new musical m^'tard-ad^ts^fXLsn^ 

audio source file 328, which acts as a Pro'cessing Guide. 

(130) For instance, musical information is extracted from two 
separate and distinct audio files: the original 300 is a flute 
passage and the second 328 is a violin passage. The original flute 
passage 300 may have played all the correct pitches in more or less 
the right timing, however, the violin passage 328 may represent a 
more dynamic and articulate performance. Therefore, comparing the two 
sets of extracted musical information 302 and 304 can impose the 
desired aspects of violin's performance onto that of the flute's in a 
new digital audio file 320. 

(131) The remaining portions of this variation on the invention are 
the same as those in the basic process (FIG. 6) , 

(132) Harmonizing 

(133) This variation of the invention seeks to harmonize monophonic 
digital audio 300. As FIG. 15 illustrates, original monophonic 
musical information 302 is compared to new polyphonic musical 
information 332; this comparison establishes pitch relationships 
between the two sources of musical information 300 and 332. 

(134) The new musical information 332 can introduce harmonic 
content far more advanced than basic, direct transpositions (major 
thirds, fifths, octaves, etc.). The new musical information 332 could 
provide harmonic content based on particular scales, modes and 
genres; in fact, the new material need not be based on the rhythmic 
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content of the original digital audio 302, thereby providing the 
possibility of more advanced polyphonic counterpoint. 

(135) Once the Comparator 306 has determined the pitch 
relationships between the original and new musical information 302 
and 332, it generates the time varying control functions 307, 309, 
311, 313, which in turn provide the necessary instructions for the 
corresponding DSP modules 314-317. 

(136) FIG. 15 illustrates the Comparator 306 actually generating 
two new layers of harmonic content, which necessitates the addition 
of a new component in this variation of the experiment: the DSP 
Mixing Module 319. The DSP Mixing Module 319 mixes the original 
digital audio 300 with the new layers down to a new digital audio 
file 320. 

(137) The number of layers to be generated and mixed is determined 
by the Comparator 306 when comparing the original musical information 
302 and new musical information 332. Additionally, if the original 
musical information 302 remains unchanged as a layer in the new 
musical information 332 then it need not be processed by the Pitch 
Shifting 314, 316 and Time Stretching 315, 317 modules; instead the 
original digital audio 300 is mixed with the new layers by the Mixing 
Module 319, 

(138) Modifying Polyphonic Source Material 

(139) FIG. 16 illustrates a variation on the invention that uses 
polyphonic material as the source for the original digital audio 300. 
The Comparator 306 examines the differences between original and new 
musical information 334, 336, both of which are polyphonic, and 
determines which voices are different. 

(140) If a particular voice is different, the comparator 306 
isolates the voice in question (by fundamental pitch) and generates 
the necessary time varying functions 338 and 340 based on its pitch 
variations. This variation of the invention requires a new DS^^-^Module 
called the Phase(2^^^codei^3;^2, which is capable of analyzing and 
resynthesizing polyphonic material. 

(141) An important aspect of the Phase Vocoder 342 is its ability 
to analyze the time varying control functions 338 and 340 from the 
Comparator 306 and determine precisely how the material should be 
resynthesized. Any voice that the Comparator determines needs to be 
processed, is done so by the Phase Vocoder 342 without disturbing the 
other voices (which do not need processing) . This is accomplished by 
only processing harmonics specific to the voice (identified by its 
fundamental pitch) that requires processing; harmonics from the other 
voices are not affected. 
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(142) After the Phase Vocoder 342 has resynthesized the modified 
voices, a new polyphonic digital audio file 320 is generated, 

(143) The attached appendix contains select source code listings of 
elements of the invention. 

(144) The invention has now been explained with reference to 
specific embodiments. Other embodiments will be apparent to those of 
ordinary skill in the art. Therefore it is not intended that these 
claims be limited, except as indicated by the appended claims. 

(1) APPENDIX 



SOURCE CODE LISTING EXCERPTS 



#include <stdio.h> #include <stdlib.h> #include <string.h> #include 
<math.h> #include 

<time . h> /*** + *** + ***********-^***^********* + ************** 
* * Definitions 

1 ttdefine SAMPLE . sub . RATE 44100 #define DURATION. sub . -- IN.sub,-- 
SECONDS 5 #define NUM. sub.— SAMPLES ( S7\MPLE . sub . -- RATE * 
DURATION. sub.-- IN. sub.-- SECONDS / 100) #define BEATS. sub. -- 
PER. sub.-- BAR 4 #define UNITS. sub.-- PER. sub. — BEAT 480 #define 
TEMPO 120 #define PITCH. sub.-- BEND. sub.-- RANGE 2 #define 
SEMI TONES. sub. — PER. sub. — BEND (4096.0 / PITCH. sub.-- BEND. sub.- - 
RANGE) #define VOLUME. sub.— STEPS 127.0 #define DYNAMIC . sub . -- RANGE 
96.0 #define INPUT. sub.— FILE. sub.— NAME "audio. in" #define 
OUTPUT. sub.-- FILE. sub.-- NAME "audio. out" #define ENTER. sub. — 
ORIGINAL. sub. — PROMPT "Enter Original Musical 

Information: .backslash. n" #define ENTER. sub. — NEW. sub. — PROMPT 
"Enter New Musical 

Information : . backslash .n" /***************** + ********* + **********-*r-jt*** 

^k-k'k'k'kie-k'k^'k'k'k -k -k D^^a TypeS 

typedef short AudioBuf f er [NUM. sub . — SAMPLES]; typedef 
struct NoteEvent [ long startTime; /* in units */ long duration; 
in units */ short noteNumber; /* 0 to 127 */ short velocity; /* 0 to 
127 */ struct NoteEvent *next; ] NoteEvent; typedef struct 
PitchBendEvent [ long startTime; /* in units */ short pitchBend; - 
8192 to 8191 */ struct PitchBendEvent *next; ] PitchBendEvent; 
typedef struct VolumeEvent [ long startTime; in units */ short 
volume; /* 0 to 127 */ struct VolumeEvent *next; ] VolumeEvent; 
typedef struct Musicalinf ormation [ NoteEvent * f irstNoteEvent ; 
PitchBendEvent *f irstpitchBendEvent; VolumeEvent *f irstVolumeEvent ; ] 
Musicallnformation; typedef struct ControlFunction [ long 
StartTime; /* in samples */ float value; /* type dependent */ float 
tempValue; for intermediary results struct ControlFunction 
*next; ] 

ControlFunction; 
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* * Global Variables * 

static AudioBuffer originalDigitalAudio; static 
AudioBuffer intermidiateAudio . sub , -- 1; static AudioBuffer 
intermidiateAudio. sub. — 2; static AudioBuffer newDigitalAudio; 
static Boolean doExtractMusicalInf ormation = FALSE; static Boolean 
doEditMusicallnformation = FALSE; Musicalinf ormation 
*originalMusical Information; Musicalinf ormation 
*newMusicalInf ormation; Control Function *pitchControl Function; 
Control Function ^gainControl Function; Control Function 
* timeControl Function; 

'k'k-^'k'k-k'k'ic'k^-k'ic -k -k PrOtOt y P e S * 

^■k-k-k-k-k-k-k-k-k-k / void ReadAudioFile char *fileName, AudioBuffer input); 

Musicalinf ormation *ExtractMusicFromAudio ( AudioBuffer audio) ; 

Musicalinf ormation *EditMusicalInf ormation ( Musicalinf ormation 

*musicallnf ormation) ; Musicalinf ormation *EnterMusical Information 

( char ^prompt) ; ControlFunction *FindPitchDif f erences 

( Musicalinf ormation *originalMusicalInf ormation, Musicalinf ormation 

^newMusicallnf ormation) ; ControlFunction *FindGainDif f erences 

( Musicalinf ormation *originalMusicalInf ormation, Musicalinf ormation 

*newMusicalInf ormation) ; ControlFunction ^FindTimingDif f erences 

( Musicalinf ormation *originalMusical Information, Musicalinf ormation 

*newMusicalInf ormation) ; void PitchShif tAudio ( ControlFunction 

^ControlFunction, AudioBuffer input, AudioBuffer output) ; void 

ChangeAudioGain ( ControlFunction ^ControlFunction, AudioBuffer input, 

AudioBuffer output); void TimeScaleAudio ( ControlFunction 

^ControlFunction, AudioBuffer input, AudioBuffer output) ; void 

WriteAudioFile ( char *fileName, AudioBuffer 

output ); 

★ -k Implementation * 

^•k-k-k-kk-k-k-k'kk-k'k-k-k^'k'k'k'k-kiri^'k-k'k-k-k-k'k'k^'k^'k^^'k'kic^'kic'ki: 

void ReadAudioFile ( char *fileName, AudioBuffer input) 
[ /* * Implementation of this function is system dependent and * 
straight-forward for someone of ordinary skill in the art. */ ] 
Musicallnformation *ExtractMusicFromAudio ( AudioBuffer audio) [ /* * 
Implementation of this function is system dependent and * straight- 
forward for someone of ordinary skill in the art. */ ] 
Musicallnformation *EditMusicalInf ormation ( Musicallnformation 
■^musicallnformation) [ /* * Implementation of this function is system 
dependent and * straight-forward for someone of ordinary skill in the 
art. It is assumed that the user will input the notes, 

pitch bend, and volume * events in ascending order and that there 
will be no duplicates. * It is straight-forward for someone of 
ordinary skill in the art to add * a sorting procedure and to remove 
duplicates. */ Musicallnformation *EnterMusicalInf ormation ( char 
^prompt) [ Musicallnformation *musicInfo; NoteEvent *newNoteEvent ; 
PitchBendEvent ^newPitchBendEvent ; VolumeEvent *newVolumeEvent; 
NoteEvent ^lastNoteEvent ; PitchBendEvent *lastPitchBendEvent ; 
VolumeEvent *lastVolumeEvent ; char input[32]; short bar; short beat; 
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short unit; short noteNumber; short velocity; short pitchBend; short 
volume; /* * Initialize */ newVolumeEvent = calloc(l, sizeof 

(VoluitieEvent) ) ; newVolumeEvent->startTime = 0; newVolumeE vent -> volume 

= 127; newPitchBendEvent = calloc(l, sizeof (PitchBendEvent) ) ; 

newpitchBendEvent->startTime = 0; newPitchBendEvent->pitchBend = 0; 

musiclnfo = calloc(l, sizeof (Musicallnformation) ) ; musiclnfo- 

>f irstVolumeEvent = lastVolumeEvent = newVolumeEvent; musiclnfo- 

>f irstPitchBendEvent = lastPitchBendEvent = newPitchBendEvent; printf 

("%s", prompt); while (1) [ printf (" .backslash. tEnter .backslash. 

"Note, .backslash. " .backslash. "Volum 

e, .backslash. " .backslash. "Bend, .backslash. " 

or .backslash. "End. backslash. " .backslash. n" ) ; scanf("%s", input); if 
(strcmp (input, "Note") == 0) [ newNoteEvent = calloc(l, sizeof 
(NoteEvent) ) ; printf (" .backslash. tEnter Start Time (bar, beat, unit) 
followed by .backslash. "; .backslash. " .backslash. n") ; scanf("%d %d %d 
%s", &bar, &beat, &unit, input); newNoteEvent->startTime = (bar - 1) 
* BEATS, sub. — PER. sub. — BAR * UNITS, sub. — PER. sub. — BEAT + (beat 
- 1) * UNITS. sub.-- PER. sub. — BEAT + unit; print f (" .backslash . tEnter 
Duration (beats, units) followed 

by .backslash. "; .backslash. " .backslash. n") ; scanf("%d %d %s", &beat, 

&unit, input); newNoteEvent->duration= beat * UNITS. sub. — PER. sub. — 

BEAT + unit; printf (" .backslash. tEnter Note Number followed 

by .backslash. "; .backslash. " .backslash. n" ) ; scanf("%d %s", -leNumber, 

input) ; newNoteEvent->noteNumber = noteNumber; printf 

(" .backslash. tEnter Velocity followed by .backslash. "; .backslash. " .b 

ackslash.n") ; scanf("%d %s", &velocity, input); newNoteEvent- 

>velocity = velocity; if (musicinf o->f irstNoteEvent == NULL) 

musicInfo->f irstNoteEvent = newNoteEvent; else lastNoteEvent->next = 

newNoteEvent; lastNoteEvent = newNoteEvent; ] else if (strcmp (input, 

"Volume") == 0) [ newVolumeEvent = calloc(l, sizeof (VolumeEvent) ) ; 

printf (" .backslash. tEnter Start Time (bar, beat, unit) followed 

by .backslash. "; .backslash. " .backslash. n") ; scanf("%d %d %d%s", &bar, 

&beat, &unit, input) ; 

(2) newVolumeEvent->startTime = (bar - 1) * BEATS. sub. — PER, sub. — 
BAR * UNITS. sub. — PER. sub. — BEAT + (beat - 1) * UNITS. sub. -- 
PER. sub.-- BEAT + unit; printf (" .backslash . tEnter Volume followed 
by .backslash. "; .backslash. " .bac kslash.n"); scanf("%d %s", 5cVolume, 
input) ; newVolumeEvent->volume = volume; lastVolumeEvent->next = 
newVolumeEvent; lastVolumeEvent = newVolumeEvent; ] else if (strcmp 

(input, "Bend") == 0) [ newPitchBendEvent = callocd, sizeof 

(PitchBendEvent)); printf (" .backslash. tEnter Start Time (bar, beat, 
unit) followed by .backslash. "; .backslash. " .backslash. n" ) ; scanf("%d 
%d %d %s", &bar, &beat, &unit, input); newPitchBendEvent->startTime = 

(bar - 1) * BEATS, sub.-- PER. sub.— BAR * UNITS, sub.— PER. sub. -- 
BEAT + (beat - 1) * UNITS. sub.-- PER. sub.— BEAT + unit; printf 

(" .backslash. tEnter Pitch Bend followed 

by .backslash. "; .backslash. " . backslash. n" ) ; scanf("%d %s", 
&pitchBend, input) newPitchBendEvent->pitchBend = pitchBend; 
lastPitchBendEvent->next ^ newPitchBendEvent; lastPitchBendEvent = 
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newPitchBendEvent ; ] else if (strcmp (input, "End") == 0) break; else 
printf ("Error •backslash. n") ; ] return musiclnfo; ] ControlFunction 
*FindPitchDif f erences ( Musicalinf ormation 
*originalMusical Information, Musicalinf ormation 

*newMusicalInformation) [ NoteEvent *originalNoteEvent ; NoteEvent 
*newNoteEvent; PitchBendEvent * currentOriginalBend; PitchBendEvent 
*currentNewBend; ControlFunction *firstValue; ControlFunction 
*currentValue; ControlFunction *newValue; float seconds; float 
semitones; float bend; long currentTime; /* * Start with notes and 
their note number. * Start at the begining of both note lists. */ 
firstValue = NULL; originalNoteEvent = originalMusicalInf ormation- 
>f irstNoteEvent; newNoteEvent = newMusicalInf ormation- 
>f irstNoteEvent; * Stop when either list is exhausted. */ while 
(originalNoteEvent != NULL && newNoteEvent != NULL) [ /* * Create a 
new point for the time-varying control function. * Give it a time in 
units and a value that is the * difference in semitones converted to 
frequency ratio. */ newValue = calloc(l, sizeof (ControlFunction) ) ; 
newValue->startTime = originalNoteEvent->startTime; semitones = 
newNoteEvent->noteNumber - originalNoteEvent->noteNumber ; newValue- 
>value = semitones; /* * Append the new point to the time-varying 
control function. */ if (firstValue == NULL) firstValue = newValue; 
else currentValue->next = newValue; currentValue = newValue; /* * 
Continue with the next pair of notes */ originalNoteEvent = 
originalNoteEvent->next ; newNoteEvent = newNoteEvent->next ; ] /* * 
Now do the pitch bend. Start at the begining of both pitch bend 
lists. */ currentOriginalBend = originalMusicalInf ormation- 
>f irstPitchBendEvent; currentNewBend = newMusicalInf ormation- 
>firstPitchBendEvent; * We start at the beginning of the function. 

currentValue = firstValue; currentTime = 0; while (1) [ /* * Check 
if we can use the current point. * If not, we have to create a new 
one and insert it in the list. if (currentValue->startTime == 
currentTime) newValue = currentValue; else [ * Create a new point 
and insert it in the list. */ newValue = callocd, sizeof 
(ControlFunction) ) ; newValue->startTime = currentTime; if (newValue- 
>startTime < f irstValue->startTime) [ newValue->value = 1.0; 
firstValue = newValue; newValue->next = currentValue; currentValue = 
newValue; ] else [ newvalue->value = currentValue->value; newvalue- 
>next ^ currentvalue->next; currentValue->next = newValue; 
currentValue = newValue; ] ] /* * Compute the difference between the 
original and the new pitch bend. * Convert to semitomes and store it 
temporarily; */ bend = currentNewBend->pitchBend - 
currentOriginalBend->pitchBend; newValue->tempValue = bend / 
SEMI TONES. sub.-- PER. sub.-- BEND; * Find the next time. * Find the 
closest to current time from the three lists: * 1, The function 
resulted from comparing note numbers * (currentValue->next) , * 2. The 
original pitch bend (currentOriginalBend- >next ) , and * 3. The new 
pitch bend (currentNewBend->next ) . */ if (currentOriginalBend->next ! 
= NULL) [ currentTime = currentOriginalBend- >next->startTime; if 
( currentNewBend->next != NULL && currentNewBend->next->startTime < 
currentTime) currentTime = currentNewBend- >next->startTime; if 



127 00 1:4343 QAPPSea cac e ea 20030724115016541 m e 



C ie%20Ne (5 7 24 03 



DOCUMENT-IDENTrFIER: US 5792971 A 



Page 34 of 47 



( currentValue->next != NULL currentValue->next->startTiiae < 
currentTime) currentTiine = currentValue->next->startTime; ] else if 
(currentNewBend->next != NULL) [ currentTime = currentNewBend->next- 
>startTime; if (currentValue->next != NULL && currentValue->next- 
>startTiine < currentTiine) currentTime = currentValue->next- 
>startTime; ] else if (currentValue->next != NULL) [ currentTime = 
currentValue->next->startTime; ] else break; /* * Advance the 
pointers only for the lists that have an element * whose start time 
matches the new found "currentTime" */ if ( currentOriginalBend- 
>next !^ NULL && currentOriginalBend->next->startTime == currentTime) 
currentOriginalBend = currentOriginalBend->next ; if ( currentNewBend- 
>next != NULL && currentNewBend->next->startTime == currentTime) 
currentNewBend = currentNewBend->next ; if ( currentValue->next != 
NULL SlSc currentValue->next->startTime == currentTime) currentValue = 
currentValue->next ; ] /* * Go over the list one more time. */ 
currentValue = firstValue; while (currentValue != NULL) [ /* * 
Convert time in units to time in samples. */ seconds = (currentValue- 
>startTime / (float) UNITS. sub.-- PER. sub.-- BEAT) * (60.0 / TEMPO); 
currentValue->startTime = (long) (seconds * SAMPLE. sub. — RATE); /* * 
Combine semitone differences from note numbers (value) * with 
semitone differences from pitch bend (tempValue) . * Convert semitones 
to frequency ratio. */ semitones = currentvalue->value + 
currentValue->temp Value; currentValue->value = pow(2^ semitones / 
12.0); currentValue = currentValue->next ; ] return firstValue; ] 
ControlFunction *FindGainDif f erences ( Musicalinf ormation 
*originalMusical Information, Musicalinf ormation 

^newMusicallnf ormation) [ NoteEvent *originalNoteEvent ; NoteEvent 
^newNoteEvent; VolumeEvent *currentOriginalVolume; VolumeEvent 
*currentNewVolume; ControlFunction ^firstValue; ControlFunction 
^currentValue; ControlFunction *newValue; float seconds; float 
velocity; float volume; long currentTime; /* * Start with notes and 
their velocity. * Start at the begining of both note lists. */ 
firstValue = NULL; originalNoteEvent = originalMusicalInf ormation- 
>f irstNoteEvent; newNoteEvent = newMusicalInf ormation- 
>firstNoteEvent; /* * Stop when either list is exhausted. */ while 
(originalNoteEvent != NULL && newNoteEvent != NULL) [ /* * Create a 
new point for the time-varying control function. * Give it a time in 
units and a value that is the * difference between velocities 
converted to dB. */ newValue = callocd, sizeof (ControlFunction) ) ; 
newValue->startTime = originalNoteEvent->startTime; velocity = 
newNoteEvent->velocity - originalNoteEvent->velocity; newValue->value 
= velocity / VOLUME. sub. — STEPS * DYNAMIC. sub. — RANGE; /* * Append 
the new point to the time-varying control function. if (firstValue 
== NULL) firstValue = newValue; else currentValue->next = newValue; 
currentValue = newValue; /* * Continue with the next pair of notes */ 
originalNoteEvent = originalNoteEvent->next ; 

(3) newNoteEvent = newNoteEvent->next ; ] /* * Now do the volume 
events. Start at the begining of both volume lists. */ 
currentOriginalVolume = originalMusicalInf ormation->firstVolumeEvent; 
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currentNewVoluitie = newMusicalInf orination->f irstVolumeEvent ; /* * We 
start at the beginning of the function. */ currentValue = firstValue; 
currentTime = 0; while (1) [ /* * Check if we can use the current 
point. * If not, we have to create a new one and insert it in the 
list. */ if (currentValue->startTime == currentTime) newValue = 
currentValue; else [ /* * Create a new point and insert it in the 
list. */ newValue = calloc(l, sizeof (ControlFunction) ) ; newValue- 
>startTime ^ currentTime; if (newValue->startTime < firstValue- 
>startTime) [ newValue->value ^ 0.0; firstValue = newValue; newValue- 
>next = currentValue; currentValue = newValue; ] else [ newValue- 
>value = currentValue->value; newValue->next = currentValue->next ; 
currentValue->next = newValue; currentValue = newValue; ] ] /* * 
Compute the difference between the original and the new volume. * 
Convert to dB store it temporarily; */ volume = currentNewVolume- 
>volume - currentOriginalVolume->volume; newValue->tempValue = 
volume / VOLUME. sub.-- STEPS * DYNAMIC . sub . -- RANGE; /* * Find the 
next time. * Find the closest to current time from the three lists: * 
1. The function resulted from comparing note numbers * (currentValue- 
>next), * 2. The original volume (currentOriginalVolume->next) , and * 
3. The new volume (currentNewVolume->next) . */ if 
(currentOriginalVolume->next != NULL) [ currentTime = 

currentOriginalVolume->next->startTime; if ( currentNewVolume->next ! 
= NULL. backslash. && currentNewVolume->next->startTime < currentTime) 
currentTime = currentNewVolume->next->startTime; if ( currentValue- 
>next != NULL && currentValue->next->startTime < currentTime) 
currentTime = currentValue->next->startTime; ] else if 
(currentNewVolume->next != NULL) [ currentTime ^ currentNewVolume- 
>next->startTime; if (currentValue->next != NULL currentValue- 
>next->startTime < currentTime) currentTime = currentValue->next- 
>startTime; ] else if (currentValue->next != NULL) [ currentTime = 
currentValue->next->startTime; ] else break; * Advance the 
pointers only for the lists that have an element * whose start time 
matches the new found "currentTime" if ( currentOriginalVolume- 
>next != NULL && currentOriginalVolume->next->startTime == 
currentTime) currentOriginalVolume = currentOriginalVolume->next ; if 
( currentNewVolume->next != NULL && currentNewVolume->next->startTime 
== currentTime) currentNewVolume = currentNewVolume->next ; if 
( currentValue->next != NULL && currentValue->next->startTime == 
currentTime) currentValue = currentValue->next ; ] /* * Go over the 
list one more time. */ currentValue = firstValue; while 
(currentValue != NULL) [ /* * Convert time in units to time in 
samples. */ seconds = (currentValue->startTime / (float) UNITS. sub. — 
PER. sub.-- BEAT) * (60.0 / TEMPO); currentValue->startTime = (long) 
(seconds * SAMPLE. sub. — RATE); /* * Combine velocity differences in 
dB (value) * with volume defferences in dB (tempValue) . * Convert 
from dB to gain multiplyer */ volume = currentValue->value + 
currentValue->tempValue; currentValue->value = pow(10, volume / 
20.0); currentValue = currentValue->next ; ] return firstValue; ] /* * 
This routine compares only the duration od the notes. * It is 
straight-forward for someone of ordinary skill in the art * to 
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compare the silences between notes too. ControlFunction 
*FindTimingDif f erences ( Musicalinf ormation 
^originalMusical Information, Musicalinf ormation 

*newMusicalInf ormation) [ NoteEvent *originalNoteEvent ; NoteEvent 
*newNoteEvent ; ControlFunction *f irstValue; ControlFunction 
*currentValue; ControlFunction *newValue; float seconds; /* * Start 
with notes and their durations. * Start at the begining of both note 
lists. */ firstValue = NULL; originalNoteEvent = 
originalMusicalInf ormation- >firstNot eEvent; newNoteEvent = 
newMusicalInf ormation->f irstNoteEvent; * Stop when either list is 
exhausted. */ while (originalNoteEvent != NULL newNoteEvent != 
NULL) [ /* * Creat^^^ a new point for the time- varying control 
function. * Give it a time in units and a value that is the ratio * 
between durations. */ newValue = calloc(l, sizeof (ControlFunction) ) ; 
newValue->startTime = originalNoteEvent->startTime; newValue->value = 
(float) newNoteEvent->duration / originalNoteEvent->dura tion; * 
Append the new point to the time-varying control function. */ if 
(firstValue == NULL) firstValue = newValue; else currentValue->next = 
newValue; currentValue = newValue; /* * Continue with the next pair 
of notes originalNoteEvent = originalNoteEvent->next ; newNoteEvent 
= newNoteEvent->next; ] * Go over the list one more time. */ 
currentValue = firstValue; while (currentValue != NULL) [ /* * 
Convert time in units to time in samples. ^/ seconds = (currentValue- 
>startTime / (float) UNITS. sub, — PER. sub. — BEAT) * (60.0 / TEMPO); 
currentValue->startTime = (long) (seconds * SAMPLE. sub. — RATE); 
currentValue = currentValue->next ; ] return firstValue; ] void 
PitchShif tAudio ( ControlFunction *c6ntrolFunction, AudioBuffer input, 
AudioBuffer output) [ /* * Implementation of this function is system 
dependent and * straight-forward for someone of ordinary skill in the 
art. ^/ ] void ChangeAudioGain ( ControlFunction *controFunctionl, 
AudioBuffer input, AudioBuffer output) [ * Implementation of this 
function is system dependent and * straight-forward for someone of 
ordinary skill in the art. */ ] void TimeScaleAudio ( ControlFunction 
^ControlFunction, AudioBuffer input, AudioBuffer output) [ /* * 
Implementation of this function is system dependent and * straight- 
forward for someone of ordinary skill in the art. ] void 
WriteAudioFile ( char *fileName, AudioBuffer output) [ /* * 
Implementation of this function is system dependent and * straight- 
forward for someone of ordinary skill in the art. */ ] void main 
(void) [ * Phase 1 * * a. Aquire "Original Digital Audio." * b. 
Extract "Original Musical Information" from "Oiginal Digital Audio" * 
or Input "Original Musical Information." ReadAudioFile ( INPUT . sub . - 
- FILE. sub. — NAME, originalDigitalAudio) ; if 
(doExtractMusical Information) originalMusicalInf ormation = 
ExtractMusicFromAudio (originalDigitalAudio ) ; else 
originalMusicallnformation = EnterMusicalInf ormation (ENTER. sub. — 
ORIGINAL. sub.-- PROMPT); /* * Phase 2 * * Edit "Original Musical 
Information" into "New Musical Information" * or Input "INew Musical 
Information. " */ if (doEditMusicalInf ormation) newMusicalInf ormation 
= EditMusicallnformation (originalMusicalInf ormati on) ; else 



127 0 0 1:4343 C: APPS ea cac e ea 20030724115016541 m e 



C ie%20Ne 6 7 24 03 



DOCUMENT-DENTIFIER: US 5792971 A 



Page 37 of 47 



(4) newMusicallnformation = EnterMusicallnformation (ENTER, sub. -- 
NEW. sub. — PROMPT); * Phase 3 * * Compare "Original Musical 
Information" and "New Musical Information" * and generate "Time- 
Varying Control Function." */ pitchControlFunction = 
FindPitchDif f erences (originalMusical Information, 

newMusicallnformation) ; A gainControlFunction = FindGainDiff erences 
(originalMusical Information, newMusicallnformation) ; 
timeControlFunction = FindTimingDiff erences 

(originalMusicalInf ormation, newMusicallnformation) ; B * Phase 4 * 

* Process "Original Digital Audio" according to "Time-Varying Control 

* Functions" and output "New Digital Audio." */ PitchShif tAudio 
(pitchControlFunction, originalDigitalAudio, intermidiateAudio . sub . — 
1) ; ChangeAudioGain (gainControlFunction, intermidiateAudio . sub . — 1, 
intermidiateAudio . sub . — 2) ; TimeScaleAudio (timeControlFunction, 
intermidiateAudio . sub . — 2, newDigitalAudio) ; WriteAudioFile 
(OUTPUT. sub.-- FILE. sub.-- NAME, newDigitalAudio); ] 



CLAIMS : 

What is claimed is: 

1. A method for obtaining a modified version of audio information 
having music-like characteristics comprising the steps of: 

a) electronically storing a sequential series of time domain 
samples representing at least a portion of said audio information; 

b) electronically storing a first set of codes corresponding to at 
least a first parameter representing said samples; 

c) electronically storing a second set of codes having a data 
structure corresponding to said first set of codes in order to permit 
comparison between said second set of codes and said first set of 
codes; thereafter 

d) electronically comparing said first set of codes and said 
second set of codes to obtain at least one time varying sequence of 
differences to be used as a control function; and 

e) electronically processing said samples under control of said 
time varying control function according to at least one DSP function 
in order to obtain said modified version of said time domain samples 
containing characteristics of said second set of codes. 

2. The method of claim 1 wherein said first set of codes and said 
second set of codes conform to the Musical Instrument Digital 
Interface (MIDI) standard. 
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3. The method of claim 1 further including the step of presenting 
a visual representation of said time varying control function to a 
user. 

4. The method of claim 1 further including the prior step of 
editing said first set of codes to obtain said second set of codes. 

5. The method of claim 1 wherein said audio information is music 
and wherein said first set of codes and said second set of codes are 
paired and wherein said first codes and said second codes comprise at 
least one of pitch, pitch bend, duration, tempo, volume, dynamic 
envelope and spectral content of a musical composition. 

6. The method of claim 1 wherein said DSP functions include at 
least one of pitch shifting, time compression, time expansion, 
amplitude changes and spectral filtering. 

7. The method of claim 6 wherein said audio information is 
polyphonic and wherein said DSP functions process at least one voice 
independently of other voices. 

8. The method of claim 1 further including the prior step of 
compressing said time domain samples and wherein said storing step a) 
comprises storing compressed representation of said time domain 
samples and wherein said processing step further includes 
decompressing. 

9. A method for obtaining a modified version of audio information 
having music-like characteristics comprising the steps of: 

a) electronically storing a sequential series of time domain 
samples representing at least a portion of said audio information; 

b) electronically storing a first set of codes corresponding to at 
least a first parameter representing said samples; 

c) electronically storing a second set of codes having a data 
structure corresponding to said first set of codes in order to permit 
comparison between said second set of codes and said first set of 
codes; thereafter 

d) electronically comparing said first set of codes and said 
second set of codes to obtain at least one time varying control 
function; and 

e) electronically processing said samples under control of said 
time varying control function according to at least one DSP function 
in order to obtain said modified version of said time domain samples 
containing characteristics of said second set of codes; 
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wherein at least one of said first set of codes and said second 
set of codes is output for use in form of at least one of standard 
music notation, piano roll form, list form, text form and strip chart 
form. 

10. A method for obtaining a modified version of audio information 
having music-like characteristics comprising the steps of: 

a) electronically storing a sequential series of time domain 
samples representing at least a portion of said audio information; 

b) electronically storing a first set of codes corresponding to at 
least a first parameter representing said samples; 

c) electronically storing a second set of codes having a data 
structure corresponding to said first set of codes in order to permit 
comparison between said second set of codes and said first set of 
codes; thereafter 

d) electronically comparing said first set of codes and said 
second set of codes to obtain at least one time varying control 
function; and 

e) electronically processing said samples under control of said 
time varying control function according to at least one DSP function 
in order to obtain said modified version of said time domain samples 
containing characteristics of said second set of codes; 

further including the prior step of quantizing said first set of 
codes to obtain said second set of codes is according to at least one 
user specified parameter. 

11. A method for obtaining a modified version of audio information 
having music-like characteristics comprising the steps of: 

a) electronically storing a sequential series of time domain 
samples representing at least a portion of said audio information: 

b) electronically storing a first set of codes corresponding to at 
least a first parameter representing said samples; 

c) electronically storing a second set of codes having a data 
structure corresponding to said first set of codes in order to permit 
comparison between said second set of codes and said first set of 
codes; thereafter 

d) electronically comparing said first set of codes and said 
second set of codes to obtain at least one time varying control 
function; and 
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e) electronically processing said samples under control of said 
time varying control function according to at least one DSP function 
in order to obtain said modified version of said time domain samples 
containing characteristics of said second set of codes; 

wherein said storing step a) comprises storing a playlist having 
events and wherein said electronically processing step e) is 
performed on at least one event in the playlist. 

12. A method for obtaining a modified version of first audio 
information having music-like characteristics comprising the steps 
of: 

a) electronically storing a first sequential series of time domain 
samples representing at least a portion of first said audio 
information; 

b) electronically storing a first set of codes corresponding to at 
least a first parameter representing said first samples; 

c) obtaining a second sequential series of time domain samples 
representing at least a portion of second audio information; 

d) electronically storing said second set of codes corresponding 
to at least a one parameter representing said second samples and 
having a data structure corresponding to said first set of codes in 
order to permit comparison between said second set of codes and said 
first set of codes; thereafter 

e) electronically comparing said first set of codes and said 
second set of codes to obtain at least one time varying sequence of 
differences to be used as a control function; and 

f) electronically processing said samples under control of said 
time varying control function according to at least one DSP function 
in order to obtain said modified version of said time domain samples 
containing characteristics of said second samples. 

13. A method for obtaining a modified version of original audio 
information having music-like characteristics comprising the steps 
of: 

a) electronically storing, in any order: 

a first series of time domain samples representing at least a 
portion of said original audio information, 

a first set of codes corresponding to at least a first time- 
varying parameter representing said first series of time domain 
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samples, and 

a second set of codes corresponding to at least a first time- 
varying parameter of a desired modified version of said first series 
of time domain samples having a data structure comparable to said 
first set of codes; 

b) electronically comparing said first set of codes and said 
second set of codes to obtain at least one time varying sequence of 
differences to be used as a control function; 

c) providing said set of samples to at least one Digital Signal 
Processing (DSP) function; 

d) providing said time varying control function to said DSP 
function; and 

e) altering said first series of time domain samples with said DSP 
function using said time varying control function in order to obtain 
a modified series of time domain samples containing characteristics 
of said second set of codes. 

14. The method of claim 13 wherein said original audio information 
is music. 

15. A method for obtaining a modified version of original audio 
information having music-like characteristics comprising the steps 
of: 

a) electronically storing, in any order: 

a first series of time domain samples representing at least a 
portion of said original audio information, 

a first set of codes corresponding to at least a first time- 
varying parameter representing said first series of time domain 
samples, and 

a second set of codes corresponding to at least a first time- 
varying parameter of a desired modified version of said first series 
of time domain samples having a data structure comparable to said 
first set of codes; 

b) electronically comparing said first set of codes and said 
second set of codes to obtain at least one time varying control 
function; 

c) providing said set of samples to at least one Digital Signal 
Processing (DSP) function; 
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d) providing said time varying control function to said DSP 
function; and 

e) altering said first series of time domain samples with said DSP 
function using said time varying control function in order to obtain 
a modified series of time domain samples containing characteristics 
of said second set of codes; 

wherein : 

a) said original audio information is monophonic and said first 
set of codes represents one voice, 

b) said desired modified version of original audio information is 
polyphonic and said second set of codes represents several voices, 

c) said comparing step b) includes comparing each of said voices 
of said second set of codes to said first set of codes to obtain two 
or more sets of time varying control functions, 

d) for each said set of time varying control functions, said step 
e) is performed where said first series of time domain samples is 
altered by said Digital Signal Processing using at least one time 
varying control function of said set of time varying control 
functions in order to obtain several modified series of time domain 
samples one for each said voice of said second set of codes, and 

e) said several modified series of time domain samples are mixed 
in order to obtain a harmonized version of said original audio 
information . 

16. The method of claim 14 wherein said original audio information 
is polyphonic and wherein said DSP function alters at least one voice 
independently of other voices. 

17. The method of claim 13 wherein said time-varying parameters 
comprise at least one of pitch, duration, loudness, brightness, 
tempo, fundamental frequency envelope, dynamics envelope, vibrato 
rate, vibrato depth, tremolo rate, tremolo depth, portamento, 
articulation, and spectral content of a musical composition. 

18. The method of claim 13 wherein said original audio information 
is voice. 

19. The method of claim 13 wherein at least one of said first set 
of codes and said second set of codes conform to the Musical 
Instrument Digital Interface (MIDI) standard. 

20. The method of claim 13 wherein said first set of codes is 
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obtained by electronically processing said first series of time 
domain samples according to at least one DSP analysis function. 

21. The method of claim 20 further including the steps of; 

a) electronically storing a third set of codes; 

b) deriving from said third set of codes at least one time varying 
analysis control function; and 

c) providing said time varying analysis control function to said 
DSP analysis function. 

22. The method of claim 21 wherein said third set of codes 
conforms to the Musical Instrument Digital Interface (MIDI) standard. 

23. The method of claim 13 wherein said second set of codes is 
obtained by editing said first set of codes. 

24. The method of claim 23 wherein said editing of said first set 
of codes is performed according to at least one of graphical editing, 
text editing, quantizing, and transposition. 

25. The method of claim 13 wherein said second set of codes is 
derived by: 

a) electronically storing a second series of time domain samples 
representing at least a portion of a second audio information; and 

b) electronically processing said second set of samples according 
to at least one analysis DSP function in order to obtain said second 
set of codes. 

26. The method of claim 25 further including the step of 
presenting a visual representation of said second series of time 
domain samples in the form of at least waveform display, sonogram 
form, and spectrogram form. 

27. The method of claim 13 wherein said DSP functions include at 
least one of pitch shifting, time compression and expansion, gain and 
spectral filtering. 

28. The method of claim 13 wherein said first series of time 
domain samples are compressed according to a data compression method 
and wherein said DSP processing step e) further includes 
decompressing said first series of time domain samples. 

29. The method of claim 13 further including compressing said 
modified series of time domain samples according to a data 
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compression method. 

30. A method for obtaining a modified version of original audio 
information having music-like characteristics comprising the steps 
of: 

a) electronically storing, in any order: 

a first series of time domain samples representing at least a 
portion of said original audio information, 

a first set of codes corresponding to at least a first time- 
varying parameter representing said first series of time domain 
samples, and 

a second set of codes corresponding to at least a first time- 
varying parameter of a desired modified version of said first series 
of time domain samples having a data structure comparable to said 
first set of codes; 

b) electronically comparing said first set of codes and said 
second set of codes to obtain at least one time varying control 
function; 

c) providing said set of samples to at least one Digital Signal 
Processing (DSP) function; 

d) providing said time varying control function to said DSP 
function; and 

e) altering said first series of time domain samples with said DSP 
function using said time varying control function in order to obtain 
a modified series of time domain samples containing characteristics 
of said second set of codes; 

wherein said first series of time domain samples is electronically 
stored as a first file on computer permanent storage and wherein said 
modified series of time domain samples is electronically stored as a 
file on computer permanent storage according to one of two methods: 

a) in a second file separate from said first file; and 

b) in said first file in order for said modified version of 
original audio information to replace said original audio 
information . 

31. A method for obtaining a modified version of original audio 
information having music-like characteristics comprising the steps 
of: 
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a) electronically storing, in any order: 

a first series of time domain samples representing at least a 
portion of said original audio information, 

a first set of codes corresponding to at least a first time- 
varying parameter representing said first series of time domain 
samples, and 

a second set of codes corresponding to at least a first time- 
varying parameter of a desired modified version of said first series 
of time domain samples having a data structure comparable to said 
first set of codes; 

b) electronically comparing said first set of codes and said 
second set of codes to obtain at least one time varying control 
function; 

c) providing said set of samples to at least one Digital Signal 
Processing (DSP) function; 

d) providing said time varying control function to said DSP 
function; and 

e) altering said first series of time domain samples with said DSP 
function using said time varying control function in order to obtain 
a modified series of time domain samples containing characteristics 
of said second set of codes; 

wherein said first series of time domain samples are derived from 
a playlist. 

32. A method for obtaining a modified version of original audio 
information having music-like characteristics comprising the steps 
of: 

a) electronically storing, in any order: 

a first series of time domain samples representing at least a 
portion of said original audio information, 

a first set of codes corresponding to at least a first time- 
varying parameter representing said first series of time domain 
samples, and 

a second set of codes corresponding to at least a first time- 
varying parameter of a desired modified version of said first series 
of time domain samples having a data structure comparable to said 
first set of codes; 
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b) electronically comparing said first set of codes and said 
second set of codes to obtain at least one time varying control 
function; 

c) providing said set of samples to at least one Digital Signal 
Processing (DSP) function; 

d) providing said time varying control function to said DSP 
function; and 

e) altering said first series of time domain samples with said DSP 
function using said time varying control function in order to obtain 
a modified series of time domain samples containing characteristics 
of said second set of codes; 

wherein at least one of said first series of time domain samples, 
said first set of codes, said second set of codes, said time varying 
control functions, and said modified series of time domain samples is 
displayed to a user of the system implementing said method. 

33. The method of claim 32 wherein at least one of said first 
series of time domain samples and of said modified series of time 
domain samples is displayed in the form of at least one of waveform 
display, sonogram form, and spectrogram form, 

34. The method of claim 32 wherein at least one of said first set 
of codes and of said second set of codes is displayed in the form of 
at least one of standard music notation form, piano roll form, list 
form, text form and strip chart form. 

35. A method for obtaining a modified version of original audio 
information having music-like characteristics comprising the steps 
of: 

a) electronically storing, in any order: 

a first series of time domain samples representing at least a 
portion of said original audio information, 

a first set of codes corresponding to at least a first time- 
varying parameter representing said first series of time domain 
samples, and 

a second set of codes corresponding to at least a first time- 
varying parameter of a desired modified version of said first series 
of time domain samples having a data structure comparable to said 
first set of codes; 

b) electronically comparing said first set of codes and said 
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second set of codes to obtain at least one time varying control 
function; 

c) providing said set of samples to at least one Digital Signal 
Processing (DSP) function; 

d) providing said time varying control function to said DSP 
function; and 

e) altering said first series of time domain samples with said DSP 
function using said time varying control function in order to obtain 
a modified series of time domain samples containing characteristics 
of said second set of codes; 

wherein said first set of codes is obtained by electronically 
processing said first series of time domain samples according to at 
least one DSP analysis function; 

further including the steps of: 

a) electronically storing a third set of codes; 

b) deriving from said third set of codes at least one time varying 
analysis control function; and 

c) providing said time varying analysis control function to said 
DSP analysis function; 

wherein said third set of codes conforms to the Musical Instrument 
Digital Interface (MIDI) standard; and 

wherein said third set of codes is displayed in the form of at 
least one of standard music notation form, piano roll form, list 
form, text form and strip chart form. 
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